Sep 11 03:53:45.600880 ip-10-0-138-179 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 03:53:46.053011 ip-10-0-138-179 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:53:46.053011 ip-10-0-138-179 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 03:53:46.053011 ip-10-0-138-179 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:53:46.053011 ip-10-0-138-179 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 03:53:46.053011 ip-10-0-138-179 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:53:46.054840 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.054746    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 03:53:46.059392 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059368    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:53:46.059392 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059388    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:53:46.059392 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059392    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:53:46.059392 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059395    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:53:46.059392 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059399    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:53:46.059392 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059402    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059405    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059409    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059413    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059418    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059421    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059424    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059426    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059429    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059432    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059434    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059437    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059440    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059442    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059446    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059449    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059451    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059454    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059457    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059460    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:53:46.059631 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059463    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059466    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059469    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059471    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059474    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059476    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059479    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059481    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059484    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059487    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059489    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059492    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059494    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059497    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059501    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059503    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059506    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059509    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059511    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059514    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:53:46.060109 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059518    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059521    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059523    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059526    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059529    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059533    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059537    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059540    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059543    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059546    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059548    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059551    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059567    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059570    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059572    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059575    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059578    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059580    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059583    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:53:46.060605 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059586    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059588    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059591    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059593    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059596    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059599    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059602    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059605    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059608    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059610    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059613    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059615    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059618    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059622    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059625    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059628    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059632    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059635    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059638    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:53:46.061063 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059641    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059644    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059647    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059649    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059652    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059654    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059657    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059659    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059662    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059665    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059667    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059670    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059672    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059675    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059677    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.059680    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061516    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061524    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061527    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:53:46.061523 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061531    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061535    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061537    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061540    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061543    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061547    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061565    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061568    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061571    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061574    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061576    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061579    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061582    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061584    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061588    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061590    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061593    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061595    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061598    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061600    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:53:46.062014 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061603    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061605    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061608    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061610    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061613    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061615    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061618    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061620    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061623    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061626    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061628    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061631    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061633    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061635    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061640    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061642    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061645    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061648    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061650    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061653    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:53:46.062544 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061655    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061658    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061660    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061663    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061665    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061668    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061671    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061674    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061677    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061679    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061682    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061684    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061687    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061692    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061695    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061698    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061700    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061703    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061706    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:53:46.063058 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061709    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061711    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061714    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061716    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061719    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061722    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061725    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061728    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061730    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061733    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061736    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061739    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061743    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061746    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061748    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061751    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061754    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061756    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061759    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:53:46.063522 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061761    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061764    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061766    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061770    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061772    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061775    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061777    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061780    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061782    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061785    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061787    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061790    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061792    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061795    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061797    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061800    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061802    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.061805    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062208    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062213    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:53:46.064002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062217    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062220    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062223    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062225    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062228    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062231    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062233    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062236    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062238    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062241    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062244    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062246    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062249    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062252    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062255    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062258    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062260    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062263    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062265    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062268    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:53:46.064505 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062270    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062273    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062276    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062280    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062284    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062287    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062290    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062292    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062295    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062297    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062300    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062302    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062306    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062309    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062311    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062314    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062317    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062320    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062323    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:53:46.065002 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062326    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062328    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062331    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062333    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062336    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062338    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062341    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062343    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062347    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062349    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062353    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062356    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062359    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062361    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062364    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062366    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062369    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062371    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062374    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062376    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:53:46.065459 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062379    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062381    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062384    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062386    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062389    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062392    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062394    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062397    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062399    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062402    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062405    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062411    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062414    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062416    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062419    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062421    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062424    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062428    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062430    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062433    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:53:46.065964 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062435    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062438    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062441    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062443    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062446    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062448    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062451    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062454    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062456    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062458    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062461    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062464    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062466    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062469    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062472    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062474    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062477    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.062480    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062573    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062607    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062613    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 03:53:46.066454 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062618    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062622    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062627    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062631    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062635    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062639    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062642    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062645    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062648    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062651    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062655    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062658    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062661    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062665    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062667    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062672    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062675    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062678    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062681    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062685    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062688    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062691    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062694    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 03:53:46.067020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062698    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062702    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062705    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062708    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062712    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062715    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062720    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062723    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062726    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062729    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062732    2582 flags.go:64] FLAG: --enable-server="true"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062736    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062740    2582 flags.go:64] FLAG: --event-burst="100"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062743    2582 flags.go:64] FLAG: --event-qps="50"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062746    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062749    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062752    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062756    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062759    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062762    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062765    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062767    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062770    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062774    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062777    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 03:53:46.067582 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062780    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062783    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062785    2582 flags.go:64] FLAG: --feature-gates=""
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062790    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062792    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062796    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062799    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062802    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062808    2582 flags.go:64] FLAG: --help="false"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062811    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062814    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062817    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062820    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062823    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062826    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062829    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062832    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062835    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062839    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062842    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062845    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062848    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062851    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062854    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 03:53:46.068212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062857    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062859    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062862    2582 flags.go:64] FLAG: --lock-file=""
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062865    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062868    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062870    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062876    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062879    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062882    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062885    2582 flags.go:64] FLAG: --logging-format="text"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062892    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062896    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062899    2582 flags.go:64] FLAG: --manifest-url=""
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062901    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062906    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062909    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062915    2582 flags.go:64] FLAG: --max-pods="110"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062918    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062920    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062923    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062927    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062930    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062932    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062935    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062944    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 03:53:46.068817 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062947    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062951    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062954    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062957    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062964    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062967    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062970    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062973    2582 flags.go:64] FLAG: --port="10250"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062976    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062979    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038c46e5464de68b7"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062982    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062985    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062988    2582 flags.go:64] FLAG: --register-node="true"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062991    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062994    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.062998    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063001    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063003    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063007    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063010    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063013    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063016    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063018    2582 flags.go:64] FLAG: --runonce="false"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063021    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063026    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 03:53:46.069429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063029    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063031    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063035    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063037    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063041    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063044    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063046    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063050    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063054    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063057    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063060    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063063    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063066    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063072    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063074    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063077    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063082    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063084    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063087    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063090    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063093    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063096    2582 flags.go:64] FLAG: --v="2"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063100    2582 flags.go:64] FLAG: --version="false"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063105    2582 flags.go:64] FLAG: --vmodule=""
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063108    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 03:53:46.070075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.063111    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063244    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063248    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063251    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063254    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063257    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063262    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063264    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063267    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063270    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063272    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063275    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063278    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063280    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063284    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063289    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063293    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063296    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063299    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:53:46.070814 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063302    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063305    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063308    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063311    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063314    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063317    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063319    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063322    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063325    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063328    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063331    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063333    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063336    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063339    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063341    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063346    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063349    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063352    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063355    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:53:46.071280 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063359    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063362    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063365    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063367    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063370    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063373    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063375    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063378    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063381    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063385    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063387    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063390    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063392    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063396    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063400    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063403    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063406    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063408    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063411    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063414    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:53:46.071762 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063416    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063419    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063421    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063424    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063426    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063429    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063432    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063434    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063437    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063439    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063442    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063444    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063448    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063450    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063453    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063456    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063459    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063461    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063464    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063467    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:53:46.072253 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063469    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063473    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063476    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063478    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063481    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063484    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063486    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063489    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063491    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063494    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063496    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063499    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063501    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063504    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063506    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063509    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063512    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063515    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063517    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063523    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:53:46.072756 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063525    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.063528    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.064142    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.071302    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.071319    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.071335    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.071341    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071397    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071403    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071406    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071409    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071412    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071416    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071418    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071421    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:53:46.073254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071424    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071427    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071429    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071432    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071435    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071437    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071440    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071442    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071445    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071448    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071450    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071453    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071456    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071459    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071462    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071465    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071471    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071474    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071476    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071479    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:53:46.073726 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071482    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071485    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071488    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071491    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071494    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071496    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071499    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071501    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071504    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071507    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071510    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071512    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071515    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071518    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071520    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071523    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071525    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071528    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071530    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:53:46.074254 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071533    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071535    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071542    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071545    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071547    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071550    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071567    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071571    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071573    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071577    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071581    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071584    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071586    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071589    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071592    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071595    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071597    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071600    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071603    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071605    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:53:46.074808 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071608    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071610    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071613    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071615    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071618    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071620    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071623    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071626    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071628    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071631    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071633    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071638    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071641    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071644    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071647    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071650    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071653    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071656    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071659    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071661    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:53:46.075299 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071664    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071666    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071669    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071674    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071677    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071681    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071684    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071687    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071690    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071693    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071695    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071698    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.071703    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071845    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071851    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:53:46.075781 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071855    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071858    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071860    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071864    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071868    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071872    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071875    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071878    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071881    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071884    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071887    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071889    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071892    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071895    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071897    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071900    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071903    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071905    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071908    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:53:46.076147 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071911    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071914    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071917    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071920    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071923    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071925    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071928    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071931    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071933    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071935    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071938    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071940    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071943    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071946    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071948    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071951    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071953    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071956    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071958    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071961    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:53:46.076613 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071965    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071968    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071971    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071974    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071977    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071980    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071982    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071985    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071988    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071990    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071993    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071995    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.071998    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072000    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072003    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072007    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072009    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072012    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072014    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072017    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:53:46.077097 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072020    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072022    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072025    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072028    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072031    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072035    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072038    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072040    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072043    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072045    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072048    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072050    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072053    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072055    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072058    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072060    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072063    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072065    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072068    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:53:46.077611 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072071    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072074    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072076    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072079    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072081    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072084    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072087    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072089    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072092    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072095    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072098    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072100    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072103    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072105    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072108    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072110    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072112    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072115    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:53:46.078066 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:46.072117    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:53:46.078506 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.072122    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:53:46.078506 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.072238    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 03:53:46.078506 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.075385    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 03:53:46.078506 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.076370    2582 server.go:1019] "Starting client certificate rotation"
Sep 11 03:53:46.078506 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.076471    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:53:46.078506 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.076530    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:53:46.079198 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.079183    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:53:46.081383 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.081326    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:53:46.096012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.095982    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 11 03:53:46.100543 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.100524    2582 log.go:25] "Validated CRI v1 image API"
Sep 11 03:53:46.102017 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.101999    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 03:53:46.105538 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.105516    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 804d52cb-3bda-4b09-a855-99ff5167dbb7:/dev/nvme0n1p4 c9249d76-12ba-4e26-a328-548496326f6c:/dev/nvme0n1p3]
Sep 11 03:53:46.105615 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.105537    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 03:53:46.106099 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.106082    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:53:46.111468 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.111351    2582 manager.go:216] Machine: {Timestamp:2026-09-11 03:53:46.109368172 +0000 UTC m=+0.373846534 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099462 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e673b666f84cafbceefb289b5d49 SystemUUID:ec21e673-b666-f84c-afbc-eefb289b5d49 BootID:3d3b8afe-053c-4850-b476-511180c0e13f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:94:43:84:f7:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:94:43:84:f7:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:8c:3d:64:f1:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 03:53:46.111468 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.111457    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 03:53:46.111614 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.111535    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 03:53:46.114067 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.114041    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 03:53:46.114204 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.114068    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-179.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 03:53:46.114923 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.114912    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 03:53:46.114956 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.114925    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 03:53:46.114956 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.114933    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:53:46.115399 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.115387    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:53:46.116160 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.116151    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 03:53:46.119808 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.119797    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:53:46.119940 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.119931    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 03:53:46.119973 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.119960    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 03:53:46.119973 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.119969    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 03:53:46.120031 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.119980    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 03:53:46.120031 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.119988    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 11 03:53:46.120031 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.119997    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 03:53:46.120478 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.120466    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:53:46.120525 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.120485    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:53:46.122769 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.122744    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 03:53:46.125393 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.125366    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 03:53:46.125474 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.125408    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 03:53:46.126910 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126894    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 03:53:46.126910 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126912    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126919    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126924    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126933    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126938    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126951    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126956    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126969    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126975    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126984    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 03:53:46.127033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.126989    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 03:53:46.127838 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.127827    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 03:53:46.127838 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.127838    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 03:53:46.130072 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.129964    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 03:53:46.130250 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.130201    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 03:53:46.131715 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.131695    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-28nsn"
Sep 11 03:53:46.132036 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.132019    2582 server.go:1268] "Started kubelet"
Sep 11 03:53:46.132139 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.132101    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 03:53:46.132978 ip-10-0-138-179 systemd[1]: Started Kubernetes Kubelet.
Sep 11 03:53:46.133181 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.132935    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 03:53:46.133181 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.133108    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 03:53:46.139780 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.139746    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 03:53:46.139954 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.139934    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-28nsn"
Sep 11 03:53:46.141636 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.141595    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 03:53:46.141958 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.141941    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 03:53:46.142848 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.142829    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-179.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 03:53:46.143732 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.142817    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-179.ec2.internal.18d427e73faa57db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-179.ec2.internal,UID:ip-10-0-138-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-179.ec2.internal,},FirstTimestamp:2026-09-11 03:53:46.131986395 +0000 UTC m=+0.396464810,LastTimestamp:2026-09-11 03:53:46.131986395 +0000 UTC m=+0.396464810,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-179.ec2.internal,}"
Sep 11 03:53:46.145874 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.145793    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 03:53:46.146000 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.145982    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.146269 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.146250    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 03:53:46.146391 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.146373    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 03:53:46.146446 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.146437    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 03:53:46.146497 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.146447    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 03:53:46.146550 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.146538    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 03:53:46.146550 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.146545    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 03:53:46.147197 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147183    2582 factory.go:55] Registering systemd factory
Sep 11 03:53:46.147197 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147200    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 11 03:53:46.147420 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147408    2582 factory.go:153] Registering CRI-O factory
Sep 11 03:53:46.147493 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147423    2582 factory.go:223] Registration of the crio container factory successfully
Sep 11 03:53:46.147493 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147477    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 03:53:46.147612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147500    2582 factory.go:103] Registering Raw factory
Sep 11 03:53:46.147612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147515    2582 manager.go:1196] Started watching for new ooms in manager
Sep 11 03:53:46.147888 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.147875    2582 manager.go:349] Starting recovery of all containers
Sep 11 03:53:46.148519 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.148492    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 03:53:46.148618 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.148583    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 03:53:46.159703 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.159685    2582 manager.go:354] Recovery completed
Sep 11 03:53:46.165687 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.165667    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:53:46.170198 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.170178    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:53:46.170314 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.170213    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:53:46.170314 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.170223    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:53:46.170910 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.170893    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 03:53:46.170910 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.170909    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 03:53:46.171018 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.170929    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:53:46.171018 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.170930    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:53:46.171646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.171632    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:53:46.171704 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.171663    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:53:46.171704 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.171674    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:53:46.172275 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.172263    2582 policy_none.go:49] "None policy: Start"
Sep 11 03:53:46.172318 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.172278    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 03:53:46.172318 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.172288    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 03:53:46.173463 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.173451    2582 policy_none.go:47] "Start"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.216291    2582 manager.go:337] "Starting Device Plugin manager"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.216344    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.216357    2582 server.go:87] "Starting device plugin registration server"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.216803    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.216819    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.216918    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.217011    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.217021    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.217638    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 03:53:46.219536 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.217697    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.266514 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.266479    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 03:53:46.267743 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.267728    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 03:53:46.267824 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.267794    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 03:53:46.267873 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.267825    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 03:53:46.267939 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.267925    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 03:53:46.270306 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.270286    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:53:46.317387 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.317263    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:53:46.318318 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.318295    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:53:46.318429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.318335    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:53:46.318429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.318346    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:53:46.318429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.318368    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.327763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.327738    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.327763 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.327766    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-179.ec2.internal\": node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.349248 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.349224    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.368997 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.368971    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal"]
Sep 11 03:53:46.369087 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.369047    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:53:46.369982 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.369960    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:53:46.370055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.369998    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:53:46.370055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.370008    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:53:46.372212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.372200    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:53:46.372348 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.372334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.372385 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.372369    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:53:46.373010 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.372994    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:53:46.373063 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.373024    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:53:46.373063 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.373035    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:53:46.373143 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.373003    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:53:46.373143 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.373101    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:53:46.373143 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.373114    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:53:46.375264 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.375247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.375356 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.375271    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:53:46.375974 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.375957    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:53:46.376046 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.375990    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:53:46.376046 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.376006    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:53:46.403373 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.403345    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-179.ec2.internal\" not found" node="ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.408008 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.407990    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-179.ec2.internal\" not found" node="ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.448415 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.448369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a884ecded236e19107d12aaa92e930e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal\" (UID: \"a884ecded236e19107d12aaa92e930e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.448415 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.448418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a884ecded236e19107d12aaa92e930e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal\" (UID: \"a884ecded236e19107d12aaa92e930e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.448653 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.448448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c473fc8668ce4f2af300af6f4bed0f91-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.ec2.internal\" (UID: \"c473fc8668ce4f2af300af6f4bed0f91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.449428 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.449409    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.549544 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.549515    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.549618 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.549601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a884ecded236e19107d12aaa92e930e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal\" (UID: \"a884ecded236e19107d12aaa92e930e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.549661 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.549630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a884ecded236e19107d12aaa92e930e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal\" (UID: \"a884ecded236e19107d12aaa92e930e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.549661 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.549648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c473fc8668ce4f2af300af6f4bed0f91-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.ec2.internal\" (UID: \"c473fc8668ce4f2af300af6f4bed0f91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.549759 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.549690    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c473fc8668ce4f2af300af6f4bed0f91-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.ec2.internal\" (UID: \"c473fc8668ce4f2af300af6f4bed0f91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.549759 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.549701    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a884ecded236e19107d12aaa92e930e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal\" (UID: \"a884ecded236e19107d12aaa92e930e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.549759 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.549731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a884ecded236e19107d12aaa92e930e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal\" (UID: \"a884ecded236e19107d12aaa92e930e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.650268 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.650196    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.706419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.706384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.711661 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:46.711462    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:46.750529 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.750499    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.850929 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.850887    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:46.951335 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:46.951268    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:47.051705 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:47.051669    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:47.075795 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.075764    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 03:53:47.076233 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.075909    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:53:47.105842 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.105816    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:53:47.152393 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.152355    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 03:48:45 +0000 UTC" deadline="2028-03-22 13:11:13.826025777 +0000 UTC"
Sep 11 03:53:47.152551 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.152414    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13401h17m26.673616755s"
Sep 11 03:53:47.152551 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:47.152370    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:47.253421 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:47.253352    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:47.296083 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:47.296043    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda884ecded236e19107d12aaa92e930e4.slice/crio-80d3e9d60b2b74128a8376da56dad5df70e932b6397c38e4ee5b27b3d562141a WatchSource:0}: Error finding container 80d3e9d60b2b74128a8376da56dad5df70e932b6397c38e4ee5b27b3d562141a: Status 404 returned error can't find the container with id 80d3e9d60b2b74128a8376da56dad5df70e932b6397c38e4ee5b27b3d562141a
Sep 11 03:53:47.296541 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:47.296516    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc473fc8668ce4f2af300af6f4bed0f91.slice/crio-3eabbdf8cb29cfaf17508f7ebb855a0b256aefa9c116ce94393bded45e2a0531 WatchSource:0}: Error finding container 3eabbdf8cb29cfaf17508f7ebb855a0b256aefa9c116ce94393bded45e2a0531: Status 404 returned error can't find the container with id 3eabbdf8cb29cfaf17508f7ebb855a0b256aefa9c116ce94393bded45e2a0531
Sep 11 03:53:47.300980 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.300964    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:53:47.354040 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:47.353999    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:47.454407 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:47.454375    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:47.465263 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.465240    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:53:47.554942 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:47.554867    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-179.ec2.internal\" not found"
Sep 11 03:53:47.564198 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.564176    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:53:47.646599 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.646565    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:47.662868 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.662846    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:53:47.663925 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.663913    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal"
Sep 11 03:53:47.673605 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:47.673588    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:53:48.120990 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.120962    2582 apiserver.go:52] "Watching apiserver"
Sep 11 03:53:48.128632 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.128601    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 03:53:48.131592 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.131542    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal","openshift-multus/multus-additional-cni-plugins-2w55b","openshift-multus/network-metrics-daemon-sntrc","openshift-ovn-kubernetes/ovnkube-node-mdgl4","kube-system/global-pull-secret-syncer-h4xhm","kube-system/konnectivity-agent-kjt6x","kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9bl9p","openshift-multus/multus-jr7h4","openshift-network-diagnostics/network-check-target-jt69z","openshift-network-operator/iptables-alerter-vv44k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw","openshift-dns/node-resolver-5n9zq","openshift-image-registry/node-ca-mljfk"]
Sep 11 03:53:48.134202 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.134176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:48.134296 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.134262    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:53:48.136270 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.136245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.138545 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.138520    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 03:53:48.138669 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.138639    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:48.138726 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.138531    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxlrv\""
Sep 11 03:53:48.138726 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.138709    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:53:48.138979 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.138961    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 03:53:48.141205 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.141185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.143493 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.143461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.143493 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.143486    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 03:53:48.143659 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.143523    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:53:48.143659 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.143543    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xt2qp\""
Sep 11 03:53:48.143659 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.143632    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 03:53:48.144812 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.143934    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 03:53:48.144812 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.143977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 03:53:48.144812 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.144021    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 03:53:48.144812 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.143977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 03:53:48.145857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.145820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.147448 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.147419    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 03:53:48.148046 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.148027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.148240 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.148222    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 03:53:48.148324 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.148026    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 03:53:48.148548 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.148525    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c74w6\""
Sep 11 03:53:48.150215 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.150197    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 03:53:48.150427 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.150408    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 03:53:48.150631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.150614    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hvmx4\""
Sep 11 03:53:48.150716 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.150635    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 03:53:48.151122 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.151062    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 03:53:48.151197 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.151135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.153437 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.153415    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 03:53:48.153532 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.153512    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dkmqp\""
Sep 11 03:53:48.153621 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.153536    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 03:53:48.153876 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.153858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.154223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.154200    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 03:53:48.156165 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.156102    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 03:53:48.156634 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.156422    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 03:53:48.156634 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.156497    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kmsfr\""
Sep 11 03:53:48.156634 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.156626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.158492 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovnkube-config\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.158671 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/070b965d-988d-4e5a-82bc-8fdac5b9ff71-hosts-file\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.158763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070b965d-988d-4e5a-82bc-8fdac5b9ff71-tmp-dir\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.158822 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-systemd\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.158822 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158787    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:53:48.158899 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-var-lib-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.158899 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-ovn\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.158899 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-node-log\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.159042 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.159042 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e11211d0-1d68-4ab1-9664-a997faf51785-dbus\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.159042 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-kubelet\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.159042 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.158984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-cnibin\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159042 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-k8s-cni-cncf-io\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159042 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-netns\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159322 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-cni-bin\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159322 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159012    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.159322 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-cni-multus\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159322 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-etc-kubernetes\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159322 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:48.159547 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8txm\" (UniqueName: \"kubernetes.io/projected/070b965d-988d-4e5a-82bc-8fdac5b9ff71-kube-api-access-f8txm\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.159547 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:48.159547 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1d14c7-a743-424b-a294-e103e0070e94-cni-binary-copy\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159547 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-hostroot\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159547 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f1d14c7-a743-424b-a294-e103e0070e94-multus-daemon-config\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159547 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-cni-netd\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6b89fb3-4c0c-4014-8137-4d3ab6f6b587-konnectivity-ca\") pod \"konnectivity-agent-kjt6x\" (UID: \"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587\") " pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwcn\" (UniqueName: \"kubernetes.io/projected/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-kube-api-access-cjwcn\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-system-cni-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159697    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qcwkd\""
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-kubelet\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-env-overrides\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovnkube-script-lib\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2vf\" (UniqueName: \"kubernetes.io/projected/3f1d14c7-a743-424b-a294-e103e0070e94-kube-api-access-pz2vf\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-run-netns\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.159857 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159835    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-log-socket\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-cni-bin\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.159995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh482\" (UniqueName: \"kubernetes.io/projected/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-kube-api-access-kh482\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e11211d0-1d68-4ab1-9664-a997faf51785-kubelet-config\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-systemd-units\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-cni-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovn-node-metrics-cert\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6b89fb3-4c0c-4014-8137-4d3ab6f6b587-agent-certs\") pod \"konnectivity-agent-kjt6x\" (UID: \"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587\") " pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-slash\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-socket-dir-parent\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-conf-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-multus-certs\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.160312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-etc-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.160975 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.160528    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:53:48.161497 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.161333    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.163034    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.163177    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.163247    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pp9lb\""
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.163770    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.164349    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s8qr2\""
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.164589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.164666    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 03:53:48.165546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.164706    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 03:53:48.184264 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.184237    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zl996"
Sep 11 03:53:48.193472 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.193416    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zl996"
Sep 11 03:53:48.247856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.247825    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 03:53:48.261228 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-slash\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-conf-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-etc-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovnkube-config\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-slash\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-conf-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-var-lib-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261394 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-etc-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-etc-selinux\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-kubelet\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-netns\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-etc-kubernetes\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261617    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-var-lib-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-kubelet\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-netns\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysconfig\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-etc-kubernetes\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysctl-conf\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-systemd\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-cni-binary-copy\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.261825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1d14c7-a743-424b-a294-e103e0070e94-cni-binary-copy\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.261945    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.261950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-hostroot\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.262015    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs podName:4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:48.762000103 +0000 UTC m=+3.026478453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs") pod "network-metrics-daemon-sntrc" (UID: "4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovnkube-config\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262119    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-hostroot\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f1d14c7-a743-424b-a294-e103e0070e94-multus-daemon-config\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-cni-netd\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-device-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwcn\" (UniqueName: \"kubernetes.io/projected/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-kube-api-access-cjwcn\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-system-cni-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-env-overrides\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-cni-netd\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-modprobe-d\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-kubernetes\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.262612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysctl-d\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-run\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-system-cni-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-cni-bin\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh482\" (UniqueName: \"kubernetes.io/projected/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-kube-api-access-kh482\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f1d14c7-a743-424b-a294-e103e0070e94-multus-daemon-config\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-cnibin\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-host\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-openvswitch\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/182b58f3-6018-4d94-bc98-0db11167be7e-etc-tuned\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262730    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-cni-bin\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262732    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1d14c7-a743-424b-a294-e103e0070e94-cni-binary-copy\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182b58f3-6018-4d94-bc98-0db11167be7e-tmp\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrf6\" (UniqueName: \"kubernetes.io/projected/5278d0cb-7178-456c-a61e-ff31d0ce8121-kube-api-access-rwrf6\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-systemd-units\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-cni-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.263459 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovn-node-metrics-cert\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-systemd-units\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262986    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-env-overrides\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.262985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-cni-dir\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wzc\" (UniqueName: \"kubernetes.io/projected/0c95d4e8-0310-4daf-a733-f10700f3bf22-kube-api-access-w9wzc\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-sys-fs\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-socket-dir-parent\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-multus-certs\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5278d0cb-7178-456c-a61e-ff31d0ce8121-iptables-alerter-script\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-multus-certs\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-multus-socket-dir-parent\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-host\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/070b965d-988d-4e5a-82bc-8fdac5b9ff71-hosts-file\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070b965d-988d-4e5a-82bc-8fdac5b9ff71-tmp-dir\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-systemd\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/070b965d-988d-4e5a-82bc-8fdac5b9ff71-hosts-file\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.264313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-ovn\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-node-log\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-systemd\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e11211d0-1d68-4ab1-9664-a997faf51785-dbus\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-run-ovn\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-node-log\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxq6\" (UniqueName: \"kubernetes.io/projected/182b58f3-6018-4d94-bc98-0db11167be7e-kube-api-access-5zxq6\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e11211d0-1d68-4ab1-9664-a997faf51785-dbus\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-serviceca\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7nr\" (UniqueName: \"kubernetes.io/projected/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-kube-api-access-2c7nr\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070b965d-988d-4e5a-82bc-8fdac5b9ff71-tmp-dir\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-cnibin\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-k8s-cni-cncf-io\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-cni-bin\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-run-k8s-cni-cncf-io\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-cni-multus\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263649    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 03:53:48.265140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-cni-bin\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6b89fb3-4c0c-4014-8137-4d3ab6f6b587-konnectivity-ca\") pod \"konnectivity-agent-kjt6x\" (UID: \"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587\") " pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-cnibin\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-cni-multus\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-registration-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhr8\" (UniqueName: \"kubernetes.io/projected/d63e387b-ad34-4c83-be3a-50c3195feb99-kube-api-access-phhr8\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8txm\" (UniqueName: \"kubernetes.io/projected/070b965d-988d-4e5a-82bc-8fdac5b9ff71-kube-api-access-f8txm\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-sys\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-lib-modules\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-kubelet\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovnkube-script-lib\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-var-lib-kubelet\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2vf\" (UniqueName: \"kubernetes.io/projected/3f1d14c7-a743-424b-a294-e103e0070e94-kube-api-access-pz2vf\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.263977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-run-netns\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-log-socket\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264021    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f1d14c7-a743-424b-a294-e103e0070e94-host-var-lib-kubelet\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.266012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e11211d0-1d68-4ab1-9664-a997faf51785-kubelet-config\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-log-socket\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-system-cni-dir\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-socket-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5278d0cb-7178-456c-a61e-ff31d0ce8121-host-slash\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264187    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6b89fb3-4c0c-4014-8137-4d3ab6f6b587-konnectivity-ca\") pod \"konnectivity-agent-kjt6x\" (UID: \"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587\") " pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-host-run-netns\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264297    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e11211d0-1d68-4ab1-9664-a997faf51785-kubelet-config\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6b89fb3-4c0c-4014-8137-4d3ab6f6b587-agent-certs\") pod \"konnectivity-agent-kjt6x\" (UID: \"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587\") " pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.264413    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.264471    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret podName:e11211d0-1d68-4ab1-9664-a997faf51785 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:48.764453549 +0000 UTC m=+3.028931916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret") pod "global-pull-secret-syncer-h4xhm" (UID: "e11211d0-1d68-4ab1-9664-a997faf51785") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:48.266719 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.264512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovnkube-script-lib\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.268027 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.268006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-ovn-node-metrics-cert\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.268164 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.268141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6b89fb3-4c0c-4014-8137-4d3ab6f6b587-agent-certs\") pod \"konnectivity-agent-kjt6x\" (UID: \"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587\") " pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.273736 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.273699    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwcn\" (UniqueName: \"kubernetes.io/projected/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-kube-api-access-cjwcn\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:48.273854 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.273779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh482\" (UniqueName: \"kubernetes.io/projected/f2ec1032-0ce5-40e5-b2de-9779b8d88a1a-kube-api-access-kh482\") pod \"ovnkube-node-mdgl4\" (UID: \"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.274003 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.273945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal" event={"ID":"c473fc8668ce4f2af300af6f4bed0f91","Type":"ContainerStarted","Data":"3eabbdf8cb29cfaf17508f7ebb855a0b256aefa9c116ce94393bded45e2a0531"}
Sep 11 03:53:48.275112 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.275093    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:53:48.275208 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.275117    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:53:48.275208 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.275132    2582 projected.go:196] Error preparing data for projected volume kube-api-access-b7vm8 for pod openshift-network-diagnostics/network-check-target-jt69z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:48.275313 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.275217    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8 podName:023d80c5-5df2-45ce-adaf-a7976e1eef03 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:48.775197823 +0000 UTC m=+3.039676181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b7vm8" (UniqueName: "kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8") pod "network-check-target-jt69z" (UID: "023d80c5-5df2-45ce-adaf-a7976e1eef03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:48.275428 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.275408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal" event={"ID":"a884ecded236e19107d12aaa92e930e4","Type":"ContainerStarted","Data":"80d3e9d60b2b74128a8376da56dad5df70e932b6397c38e4ee5b27b3d562141a"}
Sep 11 03:53:48.276180 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.276140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2vf\" (UniqueName: \"kubernetes.io/projected/3f1d14c7-a743-424b-a294-e103e0070e94-kube-api-access-pz2vf\") pod \"multus-jr7h4\" (UID: \"3f1d14c7-a743-424b-a294-e103e0070e94\") " pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.279434 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.279410    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8txm\" (UniqueName: \"kubernetes.io/projected/070b965d-988d-4e5a-82bc-8fdac5b9ff71-kube-api-access-f8txm\") pod \"node-resolver-5n9zq\" (UID: \"070b965d-988d-4e5a-82bc-8fdac5b9ff71\") " pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.365634 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.365814 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wzc\" (UniqueName: \"kubernetes.io/projected/0c95d4e8-0310-4daf-a733-f10700f3bf22-kube-api-access-w9wzc\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.365814 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-sys-fs\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.365814 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5278d0cb-7178-456c-a61e-ff31d0ce8121-iptables-alerter-script\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.365814 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-host\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.365814 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.365814 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxq6\" (UniqueName: \"kubernetes.io/projected/182b58f3-6018-4d94-bc98-0db11167be7e-kube-api-access-5zxq6\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-serviceca\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-sys-fs\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7nr\" (UniqueName: \"kubernetes.io/projected/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-kube-api-access-2c7nr\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-host\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-registration-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.365998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhr8\" (UniqueName: \"kubernetes.io/projected/d63e387b-ad34-4c83-be3a-50c3195feb99-kube-api-access-phhr8\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-sys\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366097 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-lib-modules\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-var-lib-kubelet\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-system-cni-dir\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-socket-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5278d0cb-7178-456c-a61e-ff31d0ce8121-host-slash\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-etc-selinux\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysconfig\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysctl-conf\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-systemd\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366429 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-cni-binary-copy\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366504    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5278d0cb-7178-456c-a61e-ff31d0ce8121-iptables-alerter-script\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-device-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-registration-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-device-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-sys\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-modprobe-d\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-var-lib-kubelet\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366784    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-lib-modules\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-kubernetes\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.366885 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysctl-d\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-run\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-modprobe-d\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-cnibin\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-host\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-cnibin\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/182b58f3-6018-4d94-bc98-0db11167be7e-etc-tuned\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5278d0cb-7178-456c-a61e-ff31d0ce8121-host-slash\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-etc-selinux\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182b58f3-6018-4d94-bc98-0db11167be7e-tmp\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrf6\" (UniqueName: \"kubernetes.io/projected/5278d0cb-7178-456c-a61e-ff31d0ce8121-kube-api-access-rwrf6\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysctl-d\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysconfig\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-host\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-kubernetes\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-systemd\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.367461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-socket-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.368220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367508    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-run\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.368220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.366826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c95d4e8-0310-4daf-a733-f10700f3bf22-system-cni-dir\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.368220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367568    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-serviceca\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.368220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/182b58f3-6018-4d94-bc98-0db11167be7e-etc-sysctl-conf\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.368220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.367697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d63e387b-ad34-4c83-be3a-50c3195feb99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.368220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.368014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-cni-binary-copy\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.368220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.368169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c95d4e8-0310-4daf-a733-f10700f3bf22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.370321 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.370294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182b58f3-6018-4d94-bc98-0db11167be7e-tmp\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.371317 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.371245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/182b58f3-6018-4d94-bc98-0db11167be7e-etc-tuned\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.379083 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.379054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhr8\" (UniqueName: \"kubernetes.io/projected/d63e387b-ad34-4c83-be3a-50c3195feb99-kube-api-access-phhr8\") pod \"aws-ebs-csi-driver-node-6grrw\" (UID: \"d63e387b-ad34-4c83-be3a-50c3195feb99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.379904 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.379877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wzc\" (UniqueName: \"kubernetes.io/projected/0c95d4e8-0310-4daf-a733-f10700f3bf22-kube-api-access-w9wzc\") pod \"multus-additional-cni-plugins-2w55b\" (UID: \"0c95d4e8-0310-4daf-a733-f10700f3bf22\") " pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.380084 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.379934    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7nr\" (UniqueName: \"kubernetes.io/projected/8f6d2eaa-f594-4c22-921d-31d6f641b9f0-kube-api-access-2c7nr\") pod \"node-ca-mljfk\" (UID: \"8f6d2eaa-f594-4c22-921d-31d6f641b9f0\") " pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.380245 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.380224    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxq6\" (UniqueName: \"kubernetes.io/projected/182b58f3-6018-4d94-bc98-0db11167be7e-kube-api-access-5zxq6\") pod \"tuned-9bl9p\" (UID: \"182b58f3-6018-4d94-bc98-0db11167be7e\") " pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.381088 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.381069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrf6\" (UniqueName: \"kubernetes.io/projected/5278d0cb-7178-456c-a61e-ff31d0ce8121-kube-api-access-rwrf6\") pod \"iptables-alerter-vv44k\" (UID: \"5278d0cb-7178-456c-a61e-ff31d0ce8121\") " pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.450084 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.450053    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n9zq"
Sep 11 03:53:48.459872 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.459844    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:53:48.471362 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.471342    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:53:48.475935 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.475921    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jr7h4"
Sep 11 03:53:48.483278 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.483134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mljfk"
Sep 11 03:53:48.491638 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.491622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2w55b"
Sep 11 03:53:48.498044 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.498023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bl9p"
Sep 11 03:53:48.506340 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.506321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vv44k"
Sep 11 03:53:48.512927 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.512906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw"
Sep 11 03:53:48.550852 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.550823    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:53:48.769811 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.769749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:48.769811 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.769786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:48.769959 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.769881    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:48.769959 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.769892    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:48.769959 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.769939    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret podName:e11211d0-1d68-4ab1-9664-a997faf51785 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:49.769926016 +0000 UTC m=+4.034404363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret") pod "global-pull-secret-syncer-h4xhm" (UID: "e11211d0-1d68-4ab1-9664-a997faf51785") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:48.770045 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.769965    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs podName:4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:49.769947664 +0000 UTC m=+4.034426031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs") pod "network-metrics-daemon-sntrc" (UID: "4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:48.870133 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:48.870083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:48.870280 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.870248    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:53:48.870280 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.870275    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:53:48.870395 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.870287    2582 projected.go:196] Error preparing data for projected volume kube-api-access-b7vm8 for pod openshift-network-diagnostics/network-check-target-jt69z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:48.870395 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:48.870340    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8 podName:023d80c5-5df2-45ce-adaf-a7976e1eef03 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:49.870326888 +0000 UTC m=+4.134805254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7vm8" (UniqueName: "kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8") pod "network-check-target-jt69z" (UID: "023d80c5-5df2-45ce-adaf-a7976e1eef03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:49.129624 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.129595    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070b965d_988d_4e5a_82bc_8fdac5b9ff71.slice/crio-85f157a4311128653247534253ebec9a2cf52d35a5e8d7ed88133dce7a5b6b51 WatchSource:0}: Error finding container 85f157a4311128653247534253ebec9a2cf52d35a5e8d7ed88133dce7a5b6b51: Status 404 returned error can't find the container with id 85f157a4311128653247534253ebec9a2cf52d35a5e8d7ed88133dce7a5b6b51
Sep 11 03:53:49.131125 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.131051    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63e387b_ad34_4c83_be3a_50c3195feb99.slice/crio-b4ffe241e44e77e23090f6212c2e4cee47c887f1bb6fae9b34e4d0d2d53e872a WatchSource:0}: Error finding container b4ffe241e44e77e23090f6212c2e4cee47c887f1bb6fae9b34e4d0d2d53e872a: Status 404 returned error can't find the container with id b4ffe241e44e77e23090f6212c2e4cee47c887f1bb6fae9b34e4d0d2d53e872a
Sep 11 03:53:49.133792 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.133767    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ec1032_0ce5_40e5_b2de_9779b8d88a1a.slice/crio-f0bdff2e743d06ef462ec0ab5136bc8478b0337428fb1807b088943d1d52ff99 WatchSource:0}: Error finding container f0bdff2e743d06ef462ec0ab5136bc8478b0337428fb1807b088943d1d52ff99: Status 404 returned error can't find the container with id f0bdff2e743d06ef462ec0ab5136bc8478b0337428fb1807b088943d1d52ff99
Sep 11 03:53:49.134413 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.134392    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182b58f3_6018_4d94_bc98_0db11167be7e.slice/crio-1647d6075d8f5f3d988c29865656dfef06951c09a398bfd495791c97ceb5e815 WatchSource:0}: Error finding container 1647d6075d8f5f3d988c29865656dfef06951c09a398bfd495791c97ceb5e815: Status 404 returned error can't find the container with id 1647d6075d8f5f3d988c29865656dfef06951c09a398bfd495791c97ceb5e815
Sep 11 03:53:49.136398 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.136378    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6d2eaa_f594_4c22_921d_31d6f641b9f0.slice/crio-9ae7b5deb4b330e0506a5a2ac355c0c38c49d27e3c20328c871b1087a897662a WatchSource:0}: Error finding container 9ae7b5deb4b330e0506a5a2ac355c0c38c49d27e3c20328c871b1087a897662a: Status 404 returned error can't find the container with id 9ae7b5deb4b330e0506a5a2ac355c0c38c49d27e3c20328c871b1087a897662a
Sep 11 03:53:49.136999 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.136979    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5278d0cb_7178_456c_a61e_ff31d0ce8121.slice/crio-1a59f19a7bf3abeed621ab871639ef685b687b1c23b37041969cb1282ff42d00 WatchSource:0}: Error finding container 1a59f19a7bf3abeed621ab871639ef685b687b1c23b37041969cb1282ff42d00: Status 404 returned error can't find the container with id 1a59f19a7bf3abeed621ab871639ef685b687b1c23b37041969cb1282ff42d00
Sep 11 03:53:49.137615 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.137595    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1d14c7_a743_424b_a294_e103e0070e94.slice/crio-e1088d624d93efc6a4f4eac078d27c401d3df26b45440e8ab2286d0e4828836e WatchSource:0}: Error finding container e1088d624d93efc6a4f4eac078d27c401d3df26b45440e8ab2286d0e4828836e: Status 404 returned error can't find the container with id e1088d624d93efc6a4f4eac078d27c401d3df26b45440e8ab2286d0e4828836e
Sep 11 03:53:49.138521 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.138498    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b89fb3_4c0c_4014_8137_4d3ab6f6b587.slice/crio-d18906a24c78969d5ed254ddb935da413b2d08dd74e96122a395165acca53552 WatchSource:0}: Error finding container d18906a24c78969d5ed254ddb935da413b2d08dd74e96122a395165acca53552: Status 404 returned error can't find the container with id d18906a24c78969d5ed254ddb935da413b2d08dd74e96122a395165acca53552
Sep 11 03:53:49.139484 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:53:49.139373    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c95d4e8_0310_4daf_a733_f10700f3bf22.slice/crio-cc72b52e7074f2cc36d68b17e9afbccf0b5e6fdf7da80e3940db5ffbedbe73db WatchSource:0}: Error finding container cc72b52e7074f2cc36d68b17e9afbccf0b5e6fdf7da80e3940db5ffbedbe73db: Status 404 returned error can't find the container with id cc72b52e7074f2cc36d68b17e9afbccf0b5e6fdf7da80e3940db5ffbedbe73db
Sep 11 03:53:49.197101 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.197073    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:48:47 +0000 UTC" deadline="2028-04-16 20:58:07.398477577 +0000 UTC"
Sep 11 03:53:49.197101 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.197099    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14009h4m18.201381961s"
Sep 11 03:53:49.278565 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.278521    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jr7h4" event={"ID":"3f1d14c7-a743-424b-a294-e103e0070e94","Type":"ContainerStarted","Data":"e1088d624d93efc6a4f4eac078d27c401d3df26b45440e8ab2286d0e4828836e"}
Sep 11 03:53:49.279666 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.279636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"f0bdff2e743d06ef462ec0ab5136bc8478b0337428fb1807b088943d1d52ff99"}
Sep 11 03:53:49.281503 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.281471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw" event={"ID":"d63e387b-ad34-4c83-be3a-50c3195feb99","Type":"ContainerStarted","Data":"b4ffe241e44e77e23090f6212c2e4cee47c887f1bb6fae9b34e4d0d2d53e872a"}
Sep 11 03:53:49.282596 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.282575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n9zq" event={"ID":"070b965d-988d-4e5a-82bc-8fdac5b9ff71","Type":"ContainerStarted","Data":"85f157a4311128653247534253ebec9a2cf52d35a5e8d7ed88133dce7a5b6b51"}
Sep 11 03:53:49.283734 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.283714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerStarted","Data":"cc72b52e7074f2cc36d68b17e9afbccf0b5e6fdf7da80e3940db5ffbedbe73db"}
Sep 11 03:53:49.287724 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.287693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kjt6x" event={"ID":"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587","Type":"ContainerStarted","Data":"d18906a24c78969d5ed254ddb935da413b2d08dd74e96122a395165acca53552"}
Sep 11 03:53:49.288822 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.288799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vv44k" event={"ID":"5278d0cb-7178-456c-a61e-ff31d0ce8121","Type":"ContainerStarted","Data":"1a59f19a7bf3abeed621ab871639ef685b687b1c23b37041969cb1282ff42d00"}
Sep 11 03:53:49.289920 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.289898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mljfk" event={"ID":"8f6d2eaa-f594-4c22-921d-31d6f641b9f0","Type":"ContainerStarted","Data":"9ae7b5deb4b330e0506a5a2ac355c0c38c49d27e3c20328c871b1087a897662a"}
Sep 11 03:53:49.291140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.291105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bl9p" event={"ID":"182b58f3-6018-4d94-bc98-0db11167be7e","Type":"ContainerStarted","Data":"1647d6075d8f5f3d988c29865656dfef06951c09a398bfd495791c97ceb5e815"}
Sep 11 03:53:49.293020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.293000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal" event={"ID":"c473fc8668ce4f2af300af6f4bed0f91","Type":"ContainerStarted","Data":"d4786f83c995817b424480f59b56e6ff642140071f53ab478022b893ca624c46"}
Sep 11 03:53:49.309458 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.309416    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.ec2.internal" podStartSLOduration=2.309405967 podStartE2EDuration="2.309405967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:49.308938883 +0000 UTC m=+3.573417250" watchObservedRunningTime="2026-09-11 03:53:49.309405967 +0000 UTC m=+3.573884334"
Sep 11 03:53:49.778405 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.778326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:49.778405 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.778382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:49.778643 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.778563    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:49.778698 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.778649    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs podName:4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:51.77860723 +0000 UTC m=+6.043085595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs") pod "network-metrics-daemon-sntrc" (UID: "4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:49.779345 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.779091    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:49.779345 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.779136    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret podName:e11211d0-1d68-4ab1-9664-a997faf51785 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:51.779125577 +0000 UTC m=+6.043603924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret") pod "global-pull-secret-syncer-h4xhm" (UID: "e11211d0-1d68-4ab1-9664-a997faf51785") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:49.879396 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:49.879360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:49.879593 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.879582    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:53:49.879651 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.879603    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:53:49.879651 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.879615    2582 projected.go:196] Error preparing data for projected volume kube-api-access-b7vm8 for pod openshift-network-diagnostics/network-check-target-jt69z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:49.879742 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:49.879670    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8 podName:023d80c5-5df2-45ce-adaf-a7976e1eef03 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:51.879655643 +0000 UTC m=+6.144134013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7vm8" (UniqueName: "kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8") pod "network-check-target-jt69z" (UID: "023d80c5-5df2-45ce-adaf-a7976e1eef03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:50.197918 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:50.197812    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:48:47 +0000 UTC" deadline="2028-03-13 21:40:51.681666467 +0000 UTC"
Sep 11 03:53:50.197918 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:50.197872    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13193h47m1.483798104s"
Sep 11 03:53:50.271124 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:50.271094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:50.271267 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:50.271214    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:53:50.271609 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:50.271590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:50.271690 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:50.271678    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:53:50.271749 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:50.271739    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:50.271827 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:50.271811    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:53:50.321707 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:50.321673    2582 generic.go:356] "Generic (PLEG): container finished" podID="a884ecded236e19107d12aaa92e930e4" containerID="fce06b210dd9fc4c5327467cd3261b1d4da48379e236dbc47c5dff396d7b0292" exitCode=0
Sep 11 03:53:50.322725 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:50.322699    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal" event={"ID":"a884ecded236e19107d12aaa92e930e4","Type":"ContainerDied","Data":"fce06b210dd9fc4c5327467cd3261b1d4da48379e236dbc47c5dff396d7b0292"}
Sep 11 03:53:51.357767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:51.357731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal" event={"ID":"a884ecded236e19107d12aaa92e930e4","Type":"ContainerStarted","Data":"adfbe242bed0140cc8fb0aa46010b2314a118e6b5b0da18af219fc97210aee37"}
Sep 11 03:53:51.375146 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:51.375088    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.ec2.internal" podStartSLOduration=4.375066767 podStartE2EDuration="4.375066767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:51.374707533 +0000 UTC m=+5.639185905" watchObservedRunningTime="2026-09-11 03:53:51.375066767 +0000 UTC m=+5.639545136"
Sep 11 03:53:51.797711 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:51.797609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:51.797711 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:51.797666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:51.797951 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.797813    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:51.797951 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.797876    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs podName:4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:55.797857702 +0000 UTC m=+10.062336058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs") pod "network-metrics-daemon-sntrc" (UID: "4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:51.798306 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.798260    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:51.798397 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.798315    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret podName:e11211d0-1d68-4ab1-9664-a997faf51785 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:55.798299371 +0000 UTC m=+10.062777726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret") pod "global-pull-secret-syncer-h4xhm" (UID: "e11211d0-1d68-4ab1-9664-a997faf51785") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:51.898750 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:51.898710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:51.898943 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.898926    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:53:51.899006 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.898952    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:53:51.899006 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.898964    2582 projected.go:196] Error preparing data for projected volume kube-api-access-b7vm8 for pod openshift-network-diagnostics/network-check-target-jt69z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:51.899107 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:51.899025    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8 podName:023d80c5-5df2-45ce-adaf-a7976e1eef03 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:55.899005791 +0000 UTC m=+10.163484159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7vm8" (UniqueName: "kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8") pod "network-check-target-jt69z" (UID: "023d80c5-5df2-45ce-adaf-a7976e1eef03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:52.268610 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:52.268518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:52.268756 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:52.268659    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:53:52.268892 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:52.268849    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:52.269001 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:52.268970    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:53:52.269110 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:52.269020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:52.269110 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:52.269098    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:53:54.268430 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:54.268372    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:54.268898 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:54.268509    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:53:54.268993 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:54.268962    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:54.269120 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:54.269074    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:53:54.269191 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:54.269145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:54.269283 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:54.269259    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:53:55.835587 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:55.835431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:55.835587 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:55.835487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:55.836115 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.835616    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:55.836115 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.835667    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:55.836115 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.835686    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret podName:e11211d0-1d68-4ab1-9664-a997faf51785 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:03.835663612 +0000 UTC m=+18.100141977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret") pod "global-pull-secret-syncer-h4xhm" (UID: "e11211d0-1d68-4ab1-9664-a997faf51785") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:53:55.836115 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.835764    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs podName:4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:03.835745287 +0000 UTC m=+18.100223654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs") pod "network-metrics-daemon-sntrc" (UID: "4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:53:55.935979 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:55.935901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:55.936207 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.936182    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:53:55.936207 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.936210    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:53:55.936361 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.936224    2582 projected.go:196] Error preparing data for projected volume kube-api-access-b7vm8 for pod openshift-network-diagnostics/network-check-target-jt69z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:55.936361 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:55.936345    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8 podName:023d80c5-5df2-45ce-adaf-a7976e1eef03 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:03.936323928 +0000 UTC m=+18.200802317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7vm8" (UniqueName: "kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8") pod "network-check-target-jt69z" (UID: "023d80c5-5df2-45ce-adaf-a7976e1eef03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:53:56.269895 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:56.269386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:56.269895 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:56.269491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:56.269895 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:56.269509    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:53:56.269895 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:56.269543    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:56.269895 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:56.269668    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:53:56.269895 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:56.269785    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:53:58.268935 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:58.268681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:53:58.269389 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:58.269046    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:53:58.269536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:58.269452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:53:58.269600 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:58.269547    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:53:58.269793 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:53:58.269768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:53:58.269924 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:53:58.269898    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:00.268207 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:00.268150    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:00.268715 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:00.268215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:00.268715 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:00.268244    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:00.268715 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:00.268333    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:00.268715 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:00.268442    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:00.268715 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:00.268533    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:02.268954 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:02.268918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:02.268954 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:02.268940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:02.269466 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:02.268918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:02.269466 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:02.269047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:02.269466 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:02.269129    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:02.269466 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:02.269233    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:03.892790 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:03.892755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:03.892790 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:03.892794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:03.893392 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.892911    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:54:03.893392 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.892913    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:54:03.893392 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.892964    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs podName:4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:19.892950112 +0000 UTC m=+34.157428461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs") pod "network-metrics-daemon-sntrc" (UID: "4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:54:03.893392 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.892983    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret podName:e11211d0-1d68-4ab1-9664-a997faf51785 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:19.892972431 +0000 UTC m=+34.157450780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret") pod "global-pull-secret-syncer-h4xhm" (UID: "e11211d0-1d68-4ab1-9664-a997faf51785") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:54:03.993962 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:03.993931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:03.994122 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.994060    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:54:03.994122 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.994075    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:54:03.994122 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.994084    2582 projected.go:196] Error preparing data for projected volume kube-api-access-b7vm8 for pod openshift-network-diagnostics/network-check-target-jt69z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:54:03.994242 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:03.994136    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8 podName:023d80c5-5df2-45ce-adaf-a7976e1eef03 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:19.994117749 +0000 UTC m=+34.258596115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7vm8" (UniqueName: "kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8") pod "network-check-target-jt69z" (UID: "023d80c5-5df2-45ce-adaf-a7976e1eef03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:54:04.268613 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:04.268522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:04.268763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:04.268526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:04.268763 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:04.268655    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:04.268763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:04.268523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:04.268763 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:04.268738    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:04.268988 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:04.268820    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:06.269610 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:06.269575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:06.270065 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:06.269687    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:06.270065 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:06.269693    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:06.270065 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:06.269702    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:06.270065 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:06.269805    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:06.270065 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:06.269892    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:07.393001 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.392785    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 03:54:07.393708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.393312    2582 generic.go:356] "Generic (PLEG): container finished" podID="f2ec1032-0ce5-40e5-b2de-9779b8d88a1a" containerID="fd54599968d974979837341c543b9ce989f347d3e30a12155710afe3d16809f1" exitCode=1
Sep 11 03:54:07.393708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.393374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"5feccbeeb048ce3a8ca8a978d6b42336525aeafb720818bdc094f34977cb48e2"}
Sep 11 03:54:07.393708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.393475    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"4015cb0f0b38ae14b395870013c76cd15c6afc1bdad916cf186df85b348fe22b"}
Sep 11 03:54:07.393708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.393525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"8308e860b7fbc3ec4b8884d25e9c9cf6f2414b5de73f3bad2f18a99c6d4f1e03"}
Sep 11 03:54:07.393708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.393579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"7180a53d8ae4a103c8a1f8269dff6d131b47f3ac1c76c1cb398fe0bf6023050b"}
Sep 11 03:54:07.393708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.393609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerDied","Data":"fd54599968d974979837341c543b9ce989f347d3e30a12155710afe3d16809f1"}
Sep 11 03:54:07.393708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.393638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"502560d7f4a2c8d0ec030f5643bd272041d8d3c046be5c3d5fe5655fbccbc24f"}
Sep 11 03:54:07.394787 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.394761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw" event={"ID":"d63e387b-ad34-4c83-be3a-50c3195feb99","Type":"ContainerStarted","Data":"269dff0de1a53f5eabacba40a89981829b12ad87a1ede664948b20c764392543"}
Sep 11 03:54:07.395954 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.395934    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n9zq" event={"ID":"070b965d-988d-4e5a-82bc-8fdac5b9ff71","Type":"ContainerStarted","Data":"3de72ecff35dc221268390181eacc8e10680093537d81cd2c26ca03d37eaa03f"}
Sep 11 03:54:07.397386 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.397365    2582 generic.go:356] "Generic (PLEG): container finished" podID="0c95d4e8-0310-4daf-a733-f10700f3bf22" containerID="e7a3df497630fbb87c9e792c943fe09583177a6e8c477f64bb4f1355ba5455be" exitCode=0
Sep 11 03:54:07.397474 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.397442    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerDied","Data":"e7a3df497630fbb87c9e792c943fe09583177a6e8c477f64bb4f1355ba5455be"}
Sep 11 03:54:07.398867 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.398847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kjt6x" event={"ID":"d6b89fb3-4c0c-4014-8137-4d3ab6f6b587","Type":"ContainerStarted","Data":"c81bc7947ed8f644559da4491c208cc84ef936e473c318807bcd5ec62f0f9130"}
Sep 11 03:54:07.400451 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.400427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mljfk" event={"ID":"8f6d2eaa-f594-4c22-921d-31d6f641b9f0","Type":"ContainerStarted","Data":"c81e1881c0ee3bba36aae1c3b4cf6f74b266c433f9fdec64fd7d9a03738d2770"}
Sep 11 03:54:07.401805 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.401782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bl9p" event={"ID":"182b58f3-6018-4d94-bc98-0db11167be7e","Type":"ContainerStarted","Data":"608d99190d756b86634d28659f76c5c23c6b6a7b9f64b739b54aa452a8bda58c"}
Sep 11 03:54:07.403209 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.403188    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jr7h4" event={"ID":"3f1d14c7-a743-424b-a294-e103e0070e94","Type":"ContainerStarted","Data":"cd311c9adce560e395eb38a69bcda8b2e3826559c17ed60a8dfd2476f8886bab"}
Sep 11 03:54:07.412033 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.411993    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5n9zq" podStartSLOduration=4.132329416 podStartE2EDuration="21.411980909s" totalImagesPullingTime="17.279651493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:07.411271842 +0000 UTC m=+21.675750209" watchObservedRunningTime="2026-09-11 03:54:07.411980909 +0000 UTC m=+21.676459279"
Sep 11 03:54:07.426404 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.426354    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kjt6x" podStartSLOduration=4.156463049 podStartE2EDuration="21.426342499s" totalImagesPullingTime="17.26987945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:07.425877706 +0000 UTC m=+21.690356078" watchObservedRunningTime="2026-09-11 03:54:07.426342499 +0000 UTC m=+21.690820870"
Sep 11 03:54:07.442407 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.442367    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jr7h4" podStartSLOduration=3.614062393 podStartE2EDuration="21.442348872s" totalImagesPullingTime="17.828286479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:07.442246609 +0000 UTC m=+21.706724980" watchObservedRunningTime="2026-09-11 03:54:07.442348872 +0000 UTC m=+21.706827236"
Sep 11 03:54:07.457012 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.456963    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mljfk" podStartSLOduration=4.149431788 podStartE2EDuration="21.456945616s" totalImagesPullingTime="17.307513828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:07.456598908 +0000 UTC m=+21.721077278" watchObservedRunningTime="2026-09-11 03:54:07.456945616 +0000 UTC m=+21.721423989"
Sep 11 03:54:07.501403 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.501360    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9bl9p" podStartSLOduration=4.178810589 podStartE2EDuration="21.501344695s" totalImagesPullingTime="17.322534106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:07.501005204 +0000 UTC m=+21.765483585" watchObservedRunningTime="2026-09-11 03:54:07.501344695 +0000 UTC m=+21.765823063"
Sep 11 03:54:07.812805 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:07.812776    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 03:54:08.069066 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.068976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:54:08.069527 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.069508    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:54:08.227949 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.227848    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T03:54:07.812801845Z","UUID":"c2511fb8-dbe8-4ac8-9522-db50b4c45d3f","Handler":null,"Name":"","Endpoint":""}
Sep 11 03:54:08.229693 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.229670    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 03:54:08.229818 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.229716    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 03:54:08.268391 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.268323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:08.268391 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.268336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:08.268391 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.268344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:08.268677 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:08.268442    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:08.268677 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:08.268546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:08.268677 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:08.268645    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:08.408611 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.408516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw" event={"ID":"d63e387b-ad34-4c83-be3a-50c3195feb99","Type":"ContainerStarted","Data":"0163d159f44687597501d9dfae7807c50af45391e6dfa8a028c3eb539b6ecf0b"}
Sep 11 03:54:08.410304 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.410269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vv44k" event={"ID":"5278d0cb-7178-456c-a61e-ff31d0ce8121","Type":"ContainerStarted","Data":"937f6632841477e308e218ca677fe9abb81f358c2a2e3d66ff9864c6b0d19d5b"}
Sep 11 03:54:08.426588 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:08.426521    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vv44k" podStartSLOduration=5.106647512 podStartE2EDuration="22.42650389s" totalImagesPullingTime="17.319856378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:08.426163972 +0000 UTC m=+22.690642343" watchObservedRunningTime="2026-09-11 03:54:08.42650389 +0000 UTC m=+22.690982261"
Sep 11 03:54:09.416713 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:09.416518    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 03:54:09.417164 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:09.417126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"3c3a8d11b6e623a6a92df8beffe29aa7ff66dc90ef1e4ce300d7db8e35abce8b"}
Sep 11 03:54:09.419372 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:09.419344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw" event={"ID":"d63e387b-ad34-4c83-be3a-50c3195feb99","Type":"ContainerStarted","Data":"4bc93fa7e354ec390d3f45a230802e4dfa16a61cadde0a48fe1db86f4fd6c824"}
Sep 11 03:54:09.419583 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:09.419398    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 03:54:09.436933 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:09.436841    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6grrw" podStartSLOduration=4.259661841 podStartE2EDuration="23.436824776s" totalImagesPullingTime="19.177162935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:09.436646041 +0000 UTC m=+23.701124412" watchObservedRunningTime="2026-09-11 03:54:09.436824776 +0000 UTC m=+23.701303148"
Sep 11 03:54:10.268990 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:10.268955    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:10.268990 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:10.268988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:10.269289 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:10.268955    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:10.269289 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:10.269095    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:10.269289 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:10.269160    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:10.269289 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:10.269248    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:10.819307 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:10.819279    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n9zq_070b965d-988d-4e5a-82bc-8fdac5b9ff71/dns-node-resolver/0.log"
Sep 11 03:54:12.001348 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.001140    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mljfk_8f6d2eaa-f594-4c22-921d-31d6f641b9f0/node-ca/0.log"
Sep 11 03:54:12.268128 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.268042    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:12.268128 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.268076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:12.268128 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.268046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:12.268372 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:12.268152    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:12.268372 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:12.268208    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:12.268372 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:12.268282    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:12.433346 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.433320    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 03:54:12.433701 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.433678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"d712e079b873d65314321f1a80d04a1015893b74b6f7814538a08cb0d8b68c0d"}
Sep 11 03:54:12.433914 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.433895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:54:12.434071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.434053    2582 scope.go:117] "RemoveContainer" containerID="fd54599968d974979837341c543b9ce989f347d3e30a12155710afe3d16809f1"
Sep 11 03:54:12.435708 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.435689    2582 generic.go:356] "Generic (PLEG): container finished" podID="0c95d4e8-0310-4daf-a733-f10700f3bf22" containerID="bbddb2277b6980ac914a4030540fdd8c705f2c18d38367c6af8ad29ae20f64ae" exitCode=0
Sep 11 03:54:12.435818 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.435723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerDied","Data":"bbddb2277b6980ac914a4030540fdd8c705f2c18d38367c6af8ad29ae20f64ae"}
Sep 11 03:54:12.449217 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:12.449193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:54:13.441723 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.441693    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 03:54:13.442328 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.442145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" event={"ID":"f2ec1032-0ce5-40e5-b2de-9779b8d88a1a","Type":"ContainerStarted","Data":"454f3db79530d84823fa668bc2b02bc2e340c3888164987a52bfa4b32ff4beb6"}
Sep 11 03:54:13.442427 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.442409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:54:13.442520 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.442477    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:54:13.444791 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.444752    2582 generic.go:356] "Generic (PLEG): container finished" podID="0c95d4e8-0310-4daf-a733-f10700f3bf22" containerID="a791ad432dfa25b3ef924e41658071ae585d99d7fbc4e93f71a4fd7585a2b0bf" exitCode=0
Sep 11 03:54:13.444791 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.444788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerDied","Data":"a791ad432dfa25b3ef924e41658071ae585d99d7fbc4e93f71a4fd7585a2b0bf"}
Sep 11 03:54:13.457306 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.457255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:54:13.473323 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.473284    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4" podStartSLOduration=10.112134909 podStartE2EDuration="27.473270024s" totalImagesPullingTime="17.361135115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:13.471649933 +0000 UTC m=+27.736128304" watchObservedRunningTime="2026-09-11 03:54:13.473270024 +0000 UTC m=+27.737748394"
Sep 11 03:54:13.772494 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.772400    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h4xhm"]
Sep 11 03:54:13.772640 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.772596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:13.772741 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:13.772716    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:13.775499 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.775471    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jt69z"]
Sep 11 03:54:13.775639 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.775631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:13.775772 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:13.775746    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:13.776112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.776094    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sntrc"]
Sep 11 03:54:13.776219 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:13.776200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:13.776325 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:13.776303    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:14.449062 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:14.448966    2582 generic.go:356] "Generic (PLEG): container finished" podID="0c95d4e8-0310-4daf-a733-f10700f3bf22" containerID="d82d3a8e5e40c76113e32373de3d567636066b43718a3334e490726603bc15a4" exitCode=0
Sep 11 03:54:14.449062 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:14.449050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerDied","Data":"d82d3a8e5e40c76113e32373de3d567636066b43718a3334e490726603bc15a4"}
Sep 11 03:54:14.594571 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:14.594536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:54:14.594699 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:14.594658    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 03:54:14.595050 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:14.595030    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kjt6x"
Sep 11 03:54:15.268932 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:15.268730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:15.269084 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:15.268745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:15.269084 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:15.269051    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:15.269084 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:15.269074    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:15.269084 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:15.268780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:15.269235 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:15.269153    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:17.268495 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:17.268460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:17.268888 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:17.268463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:17.268888 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:17.268589    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:17.268888 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:17.268679    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:17.268888 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:17.268463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:17.268888 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:17.268788    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:19.268209 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:19.268176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:19.268519 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:19.268278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:19.268519 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:19.268284    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:19.268519 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:19.268285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:19.268519 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:19.268362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:19.268519 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:19.268421    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:19.464986 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:19.464956    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerStarted","Data":"2a4cd5fad0f7d42d6708c892bbb0a24033d19e4774bb09b5646a3cda89c7aafb"}
Sep 11 03:54:19.913122 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:19.913086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:19.913122 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:19.913126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:19.913337 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:19.913224    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:54:19.913337 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:19.913281    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret podName:e11211d0-1d68-4ab1-9664-a997faf51785 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:51.913267544 +0000 UTC m=+66.177745897 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret") pod "global-pull-secret-syncer-h4xhm" (UID: "e11211d0-1d68-4ab1-9664-a997faf51785") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:54:19.913337 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:19.913228    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:54:19.913454 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:19.913357    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs podName:4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:51.913344796 +0000 UTC m=+66.177823145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs") pod "network-metrics-daemon-sntrc" (UID: "4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:54:20.013778 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:20.013744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:20.013930 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:20.013862    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:54:20.013930 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:20.013877    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:54:20.013930 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:20.013886    2582 projected.go:196] Error preparing data for projected volume kube-api-access-b7vm8 for pod openshift-network-diagnostics/network-check-target-jt69z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:54:20.013930 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:20.013931    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8 podName:023d80c5-5df2-45ce-adaf-a7976e1eef03 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:52.01391837 +0000 UTC m=+66.278396719 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7vm8" (UniqueName: "kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8") pod "network-check-target-jt69z" (UID: "023d80c5-5df2-45ce-adaf-a7976e1eef03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:54:20.469567 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:20.469534    2582 generic.go:356] "Generic (PLEG): container finished" podID="0c95d4e8-0310-4daf-a733-f10700f3bf22" containerID="2a4cd5fad0f7d42d6708c892bbb0a24033d19e4774bb09b5646a3cda89c7aafb" exitCode=0
Sep 11 03:54:20.469985 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:20.469588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerDied","Data":"2a4cd5fad0f7d42d6708c892bbb0a24033d19e4774bb09b5646a3cda89c7aafb"}
Sep 11 03:54:21.268848 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:21.268815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:21.269024 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:21.268929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:21.269024 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:21.268939    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:21.269024 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:21.268957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:21.269125 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:21.269047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:21.269125 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:21.269116    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:21.474681 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:21.474647    2582 generic.go:356] "Generic (PLEG): container finished" podID="0c95d4e8-0310-4daf-a733-f10700f3bf22" containerID="0284d40ffb47c32693af3e046f6e6a3532084450acc5bee9c2603809e11a37fb" exitCode=0
Sep 11 03:54:21.475043 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:21.474689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerDied","Data":"0284d40ffb47c32693af3e046f6e6a3532084450acc5bee9c2603809e11a37fb"}
Sep 11 03:54:22.480628 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:22.480444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w55b" event={"ID":"0c95d4e8-0310-4daf-a733-f10700f3bf22","Type":"ContainerStarted","Data":"77dcc49882ec03128a1ef1951022dad67bf8857b0bfc3b8d2b5e6c67f3c7d649"}
Sep 11 03:54:22.509186 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:22.509140    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2w55b" podStartSLOduration=8.454518425 podStartE2EDuration="36.509125971s" totalImagesPullingTime="28.054607546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:22.507667663 +0000 UTC m=+36.772146035" watchObservedRunningTime="2026-09-11 03:54:22.509125971 +0000 UTC m=+36.773604341"
Sep 11 03:54:23.268171 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:23.268136    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:23.268171 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:23.268171    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:23.268385 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:23.268231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:23.268385 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:23.268317    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:23.268385 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:23.268369    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:23.268483 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:23.268457    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:25.268678 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:25.268628    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:25.268678 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:25.268664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:25.268678 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:25.268677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:25.269241 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:25.268744    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:25.269241 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:25.268804    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:25.269241 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:25.268848    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:27.268123 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:27.268082    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:27.268628 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:27.268087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:27.268628 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:27.268194    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:27.268628 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:27.268089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:27.268628 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:27.268326    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:27.268628 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:27.268372    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:29.268182 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:29.268142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:29.268616 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:29.268142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:29.268616 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:29.268246    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:29.268616 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:29.268256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:29.268616 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:29.268309    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:29.268616 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:29.268389    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:31.268192 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:31.268158    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:31.268611 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:31.268198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:31.268611 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:31.268251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:31.268611 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:31.268265    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:31.268611 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:31.268321    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:31.268611 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:31.268371    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:33.268458 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:33.268423    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:33.268458 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:33.268453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:33.268895 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:33.268546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:33.268895 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:33.268592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:33.268895 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:33.268662    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:33.268895 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:33.268709    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:35.268242 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.268207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:35.268242 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.268239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:35.268674 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.268300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:35.268674 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:35.268314    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt69z" podUID="023d80c5-5df2-45ce-adaf-a7976e1eef03"
Sep 11 03:54:35.268674 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:35.268378    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h4xhm" podUID="e11211d0-1d68-4ab1-9664-a997faf51785"
Sep 11 03:54:35.268674 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:35.268465    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sntrc" podUID="4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0"
Sep 11 03:54:35.494447 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.494419    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-179.ec2.internal" event="NodeReady"
Sep 11 03:54:35.494666 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.494540    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 03:54:35.552537 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.552468    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f658l"]
Sep 11 03:54:35.578781 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.578755    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-klrjk"]
Sep 11 03:54:35.578923 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.578879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.583009 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.582987    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 03:54:35.583140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.583129    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 03:54:35.583415 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.583394    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 03:54:35.583484 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.583428    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n2mzh\""
Sep 11 03:54:35.597740 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.597722    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f658l"]
Sep 11 03:54:35.597825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.597749    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-klrjk"]
Sep 11 03:54:35.597825 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.597790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.601339 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.601319    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 03:54:35.601535 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.601517    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 03:54:35.601975 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.601634    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 03:54:35.602069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.602029    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 03:54:35.602069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.602029    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xkdhm\""
Sep 11 03:54:35.680928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.680905    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n5dbg"]
Sep 11 03:54:35.697653 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.697630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.700356 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.700329    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 03:54:35.700472 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.700404    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5zgd\""
Sep 11 03:54:35.700678 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.700664    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 03:54:35.704928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.704908    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n5dbg"]
Sep 11 03:54:35.719168 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.719151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b13312eb-ce25-4e08-96be-6f1c1c5371f3-data-volume\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.719268 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.719182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zxr\" (UniqueName: \"kubernetes.io/projected/fba0f58a-3507-49cb-9909-583301b4abfb-kube-api-access-m7zxr\") pod \"ingress-canary-f658l\" (UID: \"fba0f58a-3507-49cb-9909-583301b4abfb\") " pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.719268 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.719204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b13312eb-ce25-4e08-96be-6f1c1c5371f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.719354 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.719271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b13312eb-ce25-4e08-96be-6f1c1c5371f3-crio-socket\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.719354 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.719324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fba0f58a-3507-49cb-9909-583301b4abfb-cert\") pod \"ingress-canary-f658l\" (UID: \"fba0f58a-3507-49cb-9909-583301b4abfb\") " pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.719354 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.719349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b13312eb-ce25-4e08-96be-6f1c1c5371f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.719451 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.719385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8z9b\" (UniqueName: \"kubernetes.io/projected/b13312eb-ce25-4e08-96be-6f1c1c5371f3-kube-api-access-h8z9b\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.820765 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8z9b\" (UniqueName: \"kubernetes.io/projected/b13312eb-ce25-4e08-96be-6f1c1c5371f3-kube-api-access-h8z9b\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.820765 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b13312eb-ce25-4e08-96be-6f1c1c5371f3-data-volume\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.820765 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae43bed-2080-4a10-bb67-3c4fad5b5113-metrics-tls\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.821032 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zxr\" (UniqueName: \"kubernetes.io/projected/fba0f58a-3507-49cb-9909-583301b4abfb-kube-api-access-m7zxr\") pod \"ingress-canary-f658l\" (UID: \"fba0f58a-3507-49cb-9909-583301b4abfb\") " pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.821032 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b13312eb-ce25-4e08-96be-6f1c1c5371f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.821032 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62d9s\" (UniqueName: \"kubernetes.io/projected/1ae43bed-2080-4a10-bb67-3c4fad5b5113-kube-api-access-62d9s\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.821032 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b13312eb-ce25-4e08-96be-6f1c1c5371f3-crio-socket\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.821032 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae43bed-2080-4a10-bb67-3c4fad5b5113-config-volume\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.821032 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.820989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fba0f58a-3507-49cb-9909-583301b4abfb-cert\") pod \"ingress-canary-f658l\" (UID: \"fba0f58a-3507-49cb-9909-583301b4abfb\") " pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.821032 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.821016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b13312eb-ce25-4e08-96be-6f1c1c5371f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.821318 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.821039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae43bed-2080-4a10-bb67-3c4fad5b5113-tmp-dir\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.821318 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.821110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b13312eb-ce25-4e08-96be-6f1c1c5371f3-crio-socket\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.821318 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.821189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b13312eb-ce25-4e08-96be-6f1c1c5371f3-data-volume\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.821464 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.821443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b13312eb-ce25-4e08-96be-6f1c1c5371f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.824808 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.824782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fba0f58a-3507-49cb-9909-583301b4abfb-cert\") pod \"ingress-canary-f658l\" (UID: \"fba0f58a-3507-49cb-9909-583301b4abfb\") " pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.824910 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.824779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b13312eb-ce25-4e08-96be-6f1c1c5371f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.829626 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.829601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zxr\" (UniqueName: \"kubernetes.io/projected/fba0f58a-3507-49cb-9909-583301b4abfb-kube-api-access-m7zxr\") pod \"ingress-canary-f658l\" (UID: \"fba0f58a-3507-49cb-9909-583301b4abfb\") " pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.830140 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.830120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8z9b\" (UniqueName: \"kubernetes.io/projected/b13312eb-ce25-4e08-96be-6f1c1c5371f3-kube-api-access-h8z9b\") pod \"insights-runtime-extractor-klrjk\" (UID: \"b13312eb-ce25-4e08-96be-6f1c1c5371f3\") " pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.888099 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.888078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f658l"
Sep 11 03:54:35.908823 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.908806    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-klrjk"
Sep 11 03:54:35.921702 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.921677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae43bed-2080-4a10-bb67-3c4fad5b5113-metrics-tls\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.921795 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.921734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62d9s\" (UniqueName: \"kubernetes.io/projected/1ae43bed-2080-4a10-bb67-3c4fad5b5113-kube-api-access-62d9s\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.921795 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.921755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae43bed-2080-4a10-bb67-3c4fad5b5113-config-volume\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.921895 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.921793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae43bed-2080-4a10-bb67-3c4fad5b5113-tmp-dir\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.922130 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.922111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae43bed-2080-4a10-bb67-3c4fad5b5113-tmp-dir\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.922317 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.922299    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae43bed-2080-4a10-bb67-3c4fad5b5113-config-volume\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.923856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.923839    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae43bed-2080-4a10-bb67-3c4fad5b5113-metrics-tls\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:35.943468 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:35.943448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62d9s\" (UniqueName: \"kubernetes.io/projected/1ae43bed-2080-4a10-bb67-3c4fad5b5113-kube-api-access-62d9s\") pod \"dns-default-n5dbg\" (UID: \"1ae43bed-2080-4a10-bb67-3c4fad5b5113\") " pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:36.007199 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.006988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:36.226023 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.225993    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f658l"]
Sep 11 03:54:36.229051 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.229023    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-klrjk"]
Sep 11 03:54:36.230933 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:36.230908    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba0f58a_3507_49cb_9909_583301b4abfb.slice/crio-f68e9c1fcf7acb51c6cb74bd23b9103d5b06406e519186a71ad6d20f63ccaaab WatchSource:0}: Error finding container f68e9c1fcf7acb51c6cb74bd23b9103d5b06406e519186a71ad6d20f63ccaaab: Status 404 returned error can't find the container with id f68e9c1fcf7acb51c6cb74bd23b9103d5b06406e519186a71ad6d20f63ccaaab
Sep 11 03:54:36.231802 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:36.231777    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13312eb_ce25_4e08_96be_6f1c1c5371f3.slice/crio-a521f46ee76022715ee1aa2ec000f893f90daab18304f5a322b0dedfb2f7945f WatchSource:0}: Error finding container a521f46ee76022715ee1aa2ec000f893f90daab18304f5a322b0dedfb2f7945f: Status 404 returned error can't find the container with id a521f46ee76022715ee1aa2ec000f893f90daab18304f5a322b0dedfb2f7945f
Sep 11 03:54:36.322621 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.322529    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n5dbg"]
Sep 11 03:54:36.324915 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:36.324890    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae43bed_2080_4a10_bb67_3c4fad5b5113.slice/crio-feeb2c23e83a50ce022dc3cb9e39e29aa32560b7b30b286b8fec2635ec269be0 WatchSource:0}: Error finding container feeb2c23e83a50ce022dc3cb9e39e29aa32560b7b30b286b8fec2635ec269be0: Status 404 returned error can't find the container with id feeb2c23e83a50ce022dc3cb9e39e29aa32560b7b30b286b8fec2635ec269be0
Sep 11 03:54:36.512969 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.512932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f658l" event={"ID":"fba0f58a-3507-49cb-9909-583301b4abfb","Type":"ContainerStarted","Data":"f68e9c1fcf7acb51c6cb74bd23b9103d5b06406e519186a71ad6d20f63ccaaab"}
Sep 11 03:54:36.514112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.514084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n5dbg" event={"ID":"1ae43bed-2080-4a10-bb67-3c4fad5b5113","Type":"ContainerStarted","Data":"feeb2c23e83a50ce022dc3cb9e39e29aa32560b7b30b286b8fec2635ec269be0"}
Sep 11 03:54:36.515616 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.515587    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klrjk" event={"ID":"b13312eb-ce25-4e08-96be-6f1c1c5371f3","Type":"ContainerStarted","Data":"20f41b15a7cb0474ae6891f3cfd295485a08ae7554af1b9110f069936d8d064a"}
Sep 11 03:54:36.515788 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:36.515641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klrjk" event={"ID":"b13312eb-ce25-4e08-96be-6f1c1c5371f3","Type":"ContainerStarted","Data":"a521f46ee76022715ee1aa2ec000f893f90daab18304f5a322b0dedfb2f7945f"}
Sep 11 03:54:37.268583 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.268479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:37.268786 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.268635    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:37.268850 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.268818    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:37.272796 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.272769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rtj7z\""
Sep 11 03:54:37.272796 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.272769    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:54:37.272983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.272830    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j6dl8\""
Sep 11 03:54:37.272983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.272775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:54:37.272983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.272928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:54:37.273120 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.272988    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:54:37.521215 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:37.521121    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klrjk" event={"ID":"b13312eb-ce25-4e08-96be-6f1c1c5371f3","Type":"ContainerStarted","Data":"429df56605fc05973e5bc21fe3b5dbfc2d6a213c8630ed7e10dc7f4f0c36b448"}
Sep 11 03:54:39.529090 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.529002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f658l" event={"ID":"fba0f58a-3507-49cb-9909-583301b4abfb","Type":"ContainerStarted","Data":"2ff3965fe9bbf8d952911e6dc54a27bbc86285939c6a1a41669e3e69ed94a058"}
Sep 11 03:54:39.530840 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.530818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n5dbg" event={"ID":"1ae43bed-2080-4a10-bb67-3c4fad5b5113","Type":"ContainerStarted","Data":"74bddfa5f333bafef066f1a1fd729d277406c1525faf91a52f48c1e1546fc642"}
Sep 11 03:54:39.530915 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.530874    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:39.530915 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.530898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n5dbg" event={"ID":"1ae43bed-2080-4a10-bb67-3c4fad5b5113","Type":"ContainerStarted","Data":"b4ef949b6753c21d5a09377c60853f6a148db5d8264ce9311e2bbea80600f9ff"}
Sep 11 03:54:39.535884 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.535856    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-klrjk" event={"ID":"b13312eb-ce25-4e08-96be-6f1c1c5371f3","Type":"ContainerStarted","Data":"5a68296038fcb6167b018c324989a9d61800acbed9ab66fae66ff72b610ebb93"}
Sep 11 03:54:39.544935 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.544894    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f658l" podStartSLOduration=1.53239521 podStartE2EDuration="4.544881489s" totalImagesPullingTime="3.012486279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:39.544248261 +0000 UTC m=+53.808726632" watchObservedRunningTime="2026-09-11 03:54:39.544881489 +0000 UTC m=+53.809359859"
Sep 11 03:54:39.561710 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.561568    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-klrjk" podStartSLOduration=1.731986998 podStartE2EDuration="4.561538292s" totalImagesPullingTime="2.829551294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:54:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:39.561307476 +0000 UTC m=+53.825785857" watchObservedRunningTime="2026-09-11 03:54:39.561538292 +0000 UTC m=+53.826016664"
Sep 11 03:54:39.578212 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:39.578158    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n5dbg" podStartSLOduration=1.667398397 podStartE2EDuration="4.578144077s" totalImagesPullingTime="2.91074568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:39.577873605 +0000 UTC m=+53.842351975" watchObservedRunningTime="2026-09-11 03:54:39.578144077 +0000 UTC m=+53.842622449"
Sep 11 03:54:42.452648 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.452613    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"]
Sep 11 03:54:42.479421 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.479391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"]
Sep 11 03:54:42.479544 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.479478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"
Sep 11 03:54:42.482029 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.482009    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 03:54:42.482127 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.482006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4g566\""
Sep 11 03:54:42.671678 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.671638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/38c27ccf-4920-49bd-a92c-b0969909718e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-tqzcx\" (UID: \"38c27ccf-4920-49bd-a92c-b0969909718e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"
Sep 11 03:54:42.772359 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.772291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/38c27ccf-4920-49bd-a92c-b0969909718e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-tqzcx\" (UID: \"38c27ccf-4920-49bd-a92c-b0969909718e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"
Sep 11 03:54:42.774608 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.774584    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/38c27ccf-4920-49bd-a92c-b0969909718e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-tqzcx\" (UID: \"38c27ccf-4920-49bd-a92c-b0969909718e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"
Sep 11 03:54:42.787531 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:42.787507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"
Sep 11 03:54:43.075673 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:43.075597    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"]
Sep 11 03:54:43.078574 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:43.078531    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c27ccf_4920_49bd_a92c_b0969909718e.slice/crio-440494a19355ae4b0317abc1c4429076f781f0f9c338f15f4ab4dfd8479d1d67 WatchSource:0}: Error finding container 440494a19355ae4b0317abc1c4429076f781f0f9c338f15f4ab4dfd8479d1d67: Status 404 returned error can't find the container with id 440494a19355ae4b0317abc1c4429076f781f0f9c338f15f4ab4dfd8479d1d67
Sep 11 03:54:43.548837 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:43.548801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx" event={"ID":"38c27ccf-4920-49bd-a92c-b0969909718e","Type":"ContainerStarted","Data":"440494a19355ae4b0317abc1c4429076f781f0f9c338f15f4ab4dfd8479d1d67"}
Sep 11 03:54:44.554173 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:44.554081    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx" event={"ID":"38c27ccf-4920-49bd-a92c-b0969909718e","Type":"ContainerStarted","Data":"eb1e85b31a6f35ec6b5272dcedab00c38332f2fc9824a74cf1c115cff9822d36"}
Sep 11 03:54:44.554607 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:44.554272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"
Sep 11 03:54:44.559020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:44.558999    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx"
Sep 11 03:54:44.574009 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:44.573939    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-tqzcx" podStartSLOduration=1.484434423 podStartE2EDuration="2.573924331s" totalImagesPullingTime="1.089489908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:44.571626083 +0000 UTC m=+58.836104455" watchObservedRunningTime="2026-09-11 03:54:44.573924331 +0000 UTC m=+58.838402695"
Sep 11 03:54:45.463307 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.463280    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdgl4"
Sep 11 03:54:45.599980 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.599950    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"]
Sep 11 03:54:45.603636 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.602618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.606593 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.606549    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dq2pq\""
Sep 11 03:54:45.606710 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.606636    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 03:54:45.607741 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.607724    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 03:54:45.607947 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.607923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 03:54:45.608055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.608039    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 03:54:45.610839 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.610823    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 03:54:45.618982 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.618959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"]
Sep 11 03:54:45.625376 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.625356    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84f744bc99-n8vh8"]
Sep 11 03:54:45.627109 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.627095    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.633331 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633306    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 03:54:45.633432 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633314    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 03:54:45.633432 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633426    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 03:54:45.633540 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633454    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fvv4g\""
Sep 11 03:54:45.633540 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633491    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 03:54:45.633660 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 03:54:45.633758 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633745    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 03:54:45.633834 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.633820    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 03:54:45.643484 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.643459    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f744bc99-n8vh8"]
Sep 11 03:54:45.793682 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cfdb15c-8061-4de8-87e6-69f8ed849338-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.793682 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-serving-cert\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.793682 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9g5v\" (UniqueName: \"kubernetes.io/projected/940fdf27-b346-4804-a8c9-a362daed6fa4-kube-api-access-w9g5v\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.793908 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-oauth-config\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.793908 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-oauth-serving-cert\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.793908 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-service-ca\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.793908 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdb15c-8061-4de8-87e6-69f8ed849338-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.793908 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-console-config\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.794087 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttv6\" (UniqueName: \"kubernetes.io/projected/6cfdb15c-8061-4de8-87e6-69f8ed849338-kube-api-access-vttv6\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.794087 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.793951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cfdb15c-8061-4de8-87e6-69f8ed849338-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.894944 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.894903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-service-ca\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.895103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.894967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdb15c-8061-4de8-87e6-69f8ed849338-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.895103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.894983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-console-config\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.895103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttv6\" (UniqueName: \"kubernetes.io/projected/6cfdb15c-8061-4de8-87e6-69f8ed849338-kube-api-access-vttv6\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.895103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cfdb15c-8061-4de8-87e6-69f8ed849338-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.895103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cfdb15c-8061-4de8-87e6-69f8ed849338-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.895103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-serving-cert\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.895412 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9g5v\" (UniqueName: \"kubernetes.io/projected/940fdf27-b346-4804-a8c9-a362daed6fa4-kube-api-access-w9g5v\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.895412 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-oauth-config\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.895412 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-oauth-serving-cert\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.895868 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895839    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-service-ca\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.896049 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895859    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-console-config\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.896049 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-oauth-serving-cert\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.896238 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.895887    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cfdb15c-8061-4de8-87e6-69f8ed849338-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.897870 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.897844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-serving-cert\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.897870 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.897854    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-oauth-config\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.897870 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.897855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cfdb15c-8061-4de8-87e6-69f8ed849338-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.898025 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.897866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cfdb15c-8061-4de8-87e6-69f8ed849338-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.903937 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.903908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9g5v\" (UniqueName: \"kubernetes.io/projected/940fdf27-b346-4804-a8c9-a362daed6fa4-kube-api-access-w9g5v\") pod \"console-84f744bc99-n8vh8\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") " pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:45.904641 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.904619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttv6\" (UniqueName: \"kubernetes.io/projected/6cfdb15c-8061-4de8-87e6-69f8ed849338-kube-api-access-vttv6\") pod \"prometheus-operator-68bcd7f987-gjkfk\" (UID: \"6cfdb15c-8061-4de8-87e6-69f8ed849338\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.912445 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.912429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"
Sep 11 03:54:45.935551 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:45.935519    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:46.199903 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:46.199873    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk"]
Sep 11 03:54:46.203181 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:46.203158    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfdb15c_8061_4de8_87e6_69f8ed849338.slice/crio-52defa4ac4a503ec6583990c0d562e8ad28a242ef008562091e96dbbe8ccee49 WatchSource:0}: Error finding container 52defa4ac4a503ec6583990c0d562e8ad28a242ef008562091e96dbbe8ccee49: Status 404 returned error can't find the container with id 52defa4ac4a503ec6583990c0d562e8ad28a242ef008562091e96dbbe8ccee49
Sep 11 03:54:46.215816 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:46.215792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f744bc99-n8vh8"]
Sep 11 03:54:46.218862 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:46.218840    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940fdf27_b346_4804_a8c9_a362daed6fa4.slice/crio-b08c98ac99117c396b2f96e21d3b207d97c212471949c96bdf59a94bcec3a08f WatchSource:0}: Error finding container b08c98ac99117c396b2f96e21d3b207d97c212471949c96bdf59a94bcec3a08f: Status 404 returned error can't find the container with id b08c98ac99117c396b2f96e21d3b207d97c212471949c96bdf59a94bcec3a08f
Sep 11 03:54:46.561386 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:46.561282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk" event={"ID":"6cfdb15c-8061-4de8-87e6-69f8ed849338","Type":"ContainerStarted","Data":"52defa4ac4a503ec6583990c0d562e8ad28a242ef008562091e96dbbe8ccee49"}
Sep 11 03:54:46.562475 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:46.562451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f744bc99-n8vh8" event={"ID":"940fdf27-b346-4804-a8c9-a362daed6fa4","Type":"ContainerStarted","Data":"b08c98ac99117c396b2f96e21d3b207d97c212471949c96bdf59a94bcec3a08f"}
Sep 11 03:54:48.571818 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:48.571779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk" event={"ID":"6cfdb15c-8061-4de8-87e6-69f8ed849338","Type":"ContainerStarted","Data":"9656cddc1f5bf7a7e9b0b9011d020b2381625640a3685ac167a0f74186a7ca6c"}
Sep 11 03:54:48.572247 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:48.571840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk" event={"ID":"6cfdb15c-8061-4de8-87e6-69f8ed849338","Type":"ContainerStarted","Data":"f3b20344ea415309940714fc491ee69d52448e418a17f33a27648f7c20cd4c4d"}
Sep 11 03:54:48.594607 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:48.594541    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gjkfk" podStartSLOduration=2.141305599 podStartE2EDuration="3.594523118s" totalImagesPullingTime="1.453217519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:48.592597405 +0000 UTC m=+62.857075778" watchObservedRunningTime="2026-09-11 03:54:48.594523118 +0000 UTC m=+62.859001489"
Sep 11 03:54:49.541572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:49.541521    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n5dbg"
Sep 11 03:54:49.583259 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:49.583221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f744bc99-n8vh8" event={"ID":"940fdf27-b346-4804-a8c9-a362daed6fa4","Type":"ContainerStarted","Data":"b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624"}
Sep 11 03:54:49.602279 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:49.602214    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84f744bc99-n8vh8" podStartSLOduration=1.72115167 podStartE2EDuration="4.602193182s" totalImagesPullingTime="2.881041512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:49.60094181 +0000 UTC m=+63.865420179" watchObservedRunningTime="2026-09-11 03:54:49.602193182 +0000 UTC m=+63.866671554"
Sep 11 03:54:50.964232 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.964196    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"]
Sep 11 03:54:50.966314 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.966298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:50.968802 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.968779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 03:54:50.969099 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.969084    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 03:54:50.969624 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.969609    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-28vlf\""
Sep 11 03:54:50.978992 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.978969    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"]
Sep 11 03:54:50.983092 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.983067    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"]
Sep 11 03:54:50.985106 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.985087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:50.987690 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.987668    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 03:54:50.987899 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.987886    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 03:54:50.988289 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.988274    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r26vt\""
Sep 11 03:54:50.988773 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.988757    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 03:54:50.995965 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.995945    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-49wg9"]
Sep 11 03:54:50.997844 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:50.997830    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.000041 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.000020    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zn7tg\""
Sep 11 03:54:51.000145 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.000039    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 03:54:51.000330 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.000314    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 03:54:51.000474 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.000446    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"]
Sep 11 03:54:51.000755 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.000738    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 03:54:51.034040 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxc8\" (UniqueName: \"kubernetes.io/projected/66968c6b-0d96-4040-b342-1de06758e994-kube-api-access-lqxc8\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034172 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66968c6b-0d96-4040-b342-1de06758e994-metrics-client-ca\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034172 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrgd\" (UniqueName: \"kubernetes.io/projected/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-api-access-brrgd\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.034172 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-wtmp\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034172 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.034317 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-tls\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034317 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034317 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034220    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.034317 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.034440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-textfile\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.034440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-root\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.034572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wrz\" (UniqueName: \"kubernetes.io/projected/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-kube-api-access-s4wrz\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.034572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-sys\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.034572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-accelerators-collector-config\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.034749 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.034749 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.034596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.135116 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-accelerators-collector-config\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.135116 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.135331 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.135331 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxc8\" (UniqueName: \"kubernetes.io/projected/66968c6b-0d96-4040-b342-1de06758e994-kube-api-access-lqxc8\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.135331 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:51.135244    2582 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 03:54:51.135331 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66968c6b-0d96-4040-b342-1de06758e994-metrics-client-ca\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.135331 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:51.135326    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-tls podName:0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:51.635306553 +0000 UTC m=+65.899784916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-zbjgj" (UID: "0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8") : secret "kube-state-metrics-tls" not found
Sep 11 03:54:51.135587 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrgd\" (UniqueName: \"kubernetes.io/projected/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-api-access-brrgd\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.135587 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-wtmp\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.135587 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.135587 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-tls\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.135587 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:51.135585    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 03:54:51.135828 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:51.135645    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-tls podName:66968c6b-0d96-4040-b342-1de06758e994 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:51.63563053 +0000 UTC m=+65.900108897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-tls") pod "node-exporter-49wg9" (UID: "66968c6b-0d96-4040-b342-1de06758e994") : secret "node-exporter-tls" not found
Sep 11 03:54:51.135828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.135828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-wtmp\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.135828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.135828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-textfile\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-accelerators-collector-config\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-root\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-root\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.135853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66968c6b-0d96-4040-b342-1de06758e994-metrics-client-ca\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wrz\" (UniqueName: \"kubernetes.io/projected/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-kube-api-access-s4wrz\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.136069 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-sys\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136610 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.136610 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-textfile\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136610 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:51.136203    2582 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 03:54:51.136610 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:51.136243    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-tls podName:7164ec5f-d1cf-41e7-84f1-28405bdaaaaf nodeName:}" failed. No retries permitted until 2026-09-11 03:54:51.636230173 +0000 UTC m=+65.900708526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-bsg64" (UID: "7164ec5f-d1cf-41e7-84f1-28405bdaaaaf") : secret "openshift-state-metrics-tls" not found
Sep 11 03:54:51.136610 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66968c6b-0d96-4040-b342-1de06758e994-sys\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.136610 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.136815 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.136615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.138111 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.138095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.138492 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.138465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.138630 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.138609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.146413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.146394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxc8\" (UniqueName: \"kubernetes.io/projected/66968c6b-0d96-4040-b342-1de06758e994-kube-api-access-lqxc8\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.148756 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.148735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrgd\" (UniqueName: \"kubernetes.io/projected/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-api-access-brrgd\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.148877 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.148740    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wrz\" (UniqueName: \"kubernetes.io/projected/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-kube-api-access-s4wrz\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.575155 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.575123    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67b47b569c-4qrkh"]
Sep 11 03:54:51.578041 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.578026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.600847 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.600816    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 03:54:51.607306 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.607279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b47b569c-4qrkh"]
Sep 11 03:54:51.640839 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.640813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-serving-cert\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.640957 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.640846    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lt8g\" (UniqueName: \"kubernetes.io/projected/a7a3c269-b646-464a-a4a4-2780be3c3c31-kube-api-access-2lt8g\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.640957 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.640865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-config\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.640957 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.640886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-oauth-serving-cert\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.641055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.640962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-trusted-ca-bundle\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.641055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.641002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.641055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.641020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.641055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.641048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-service-ca\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.641170 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.641071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-tls\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.641170 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.641092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-oauth-config\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.643466 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.643441    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zbjgj\" (UID: \"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.643466 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.643451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66968c6b-0d96-4040-b342-1de06758e994-node-exporter-tls\") pod \"node-exporter-49wg9\" (UID: \"66968c6b-0d96-4040-b342-1de06758e994\") " pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.643618 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.643510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7164ec5f-d1cf-41e7-84f1-28405bdaaaaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-bsg64\" (UID: \"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.741611 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.741574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-service-ca\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.741787 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.741631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-oauth-config\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.741787 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.741681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-serving-cert\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.741787 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.741704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lt8g\" (UniqueName: \"kubernetes.io/projected/a7a3c269-b646-464a-a4a4-2780be3c3c31-kube-api-access-2lt8g\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.741787 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.741728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-config\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.741787 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.741760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-oauth-serving-cert\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.742014 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.741803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-trusted-ca-bundle\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.742457 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.742396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-config\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.742457 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.742447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-service-ca\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.742616 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.742489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-oauth-serving-cert\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.742653 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.742628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-trusted-ca-bundle\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.744045 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.744025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-serving-cert\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.744103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.744078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-oauth-config\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.753964 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.753945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lt8g\" (UniqueName: \"kubernetes.io/projected/a7a3c269-b646-464a-a4a4-2780be3c3c31-kube-api-access-2lt8g\") pod \"console-67b47b569c-4qrkh\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") " pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.874652 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.874546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"
Sep 11 03:54:51.887296 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.887263    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:54:51.893456 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.893440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"
Sep 11 03:54:51.907745 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.907716    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-49wg9"
Sep 11 03:54:51.943896 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.943389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:51.943896 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.943500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:51.946359 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.946331    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:54:51.947914 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.947640    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:54:51.956730 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.956207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e11211d0-1d68-4ab1-9664-a997faf51785-original-pull-secret\") pod \"global-pull-secret-syncer-h4xhm\" (UID: \"e11211d0-1d68-4ab1-9664-a997faf51785\") " pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:51.957008 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.956874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0-metrics-certs\") pod \"network-metrics-daemon-sntrc\" (UID: \"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0\") " pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:51.981842 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.981810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h4xhm"
Sep 11 03:54:51.999164 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:51.999123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rtj7z\""
Sep 11 03:54:52.007327 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.007303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sntrc"
Sep 11 03:54:52.033692 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.033664    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:54:52.041501 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.040908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.043853 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.043528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 03:54:52.044156 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044126    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 03:54:52.044156 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044149    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 03:54:52.044390 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044352    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 03:54:52.044390 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044352    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 03:54:52.044390 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:52.044597 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044487    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 03:54:52.044860 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044840    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 03:54:52.044934 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 03:54:52.044934 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.044884    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 03:54:52.045132 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.045114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6wmbg\""
Sep 11 03:54:52.054846 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.053898    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:54:52.055798 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.055773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:54:52.056924 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.056908    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:54:52.071902 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.071881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vm8\" (UniqueName: \"kubernetes.io/projected/023d80c5-5df2-45ce-adaf-a7976e1eef03-kube-api-access-b7vm8\") pod \"network-check-target-jt69z\" (UID: \"023d80c5-5df2-45ce-adaf-a7976e1eef03\") " pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:52.145685 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-out\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145685 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145685 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145916 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145916 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145916 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145916 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145916 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.145916 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.146134 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.145942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-web-config\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.146134 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.146008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.146134 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.146041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmbl\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-kube-api-access-xvmbl\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.146134 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.146057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.157611 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.157440    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64"]
Sep 11 03:54:52.202297 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.202276    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b47b569c-4qrkh"]
Sep 11 03:54:52.207621 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.207530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj"]
Sep 11 03:54:52.211778 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:52.211740    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a3c269_b646_464a_a4a4_2780be3c3c31.slice/crio-f87ceb53e59badc48aeaeb756d29eb424edc05578cbbe72b7f2e47d8feaa281d WatchSource:0}: Error finding container f87ceb53e59badc48aeaeb756d29eb424edc05578cbbe72b7f2e47d8feaa281d: Status 404 returned error can't find the container with id f87ceb53e59badc48aeaeb756d29eb424edc05578cbbe72b7f2e47d8feaa281d
Sep 11 03:54:52.211980 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:52.211958    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8b3001_c5a8_48fb_9eec_cb3ca48cd9f8.slice/crio-c47aadaf6e76c9f185b35f92c72acfd314466a6a55ba5a71c3792e0f29b18f59 WatchSource:0}: Error finding container c47aadaf6e76c9f185b35f92c72acfd314466a6a55ba5a71c3792e0f29b18f59: Status 404 returned error can't find the container with id c47aadaf6e76c9f185b35f92c72acfd314466a6a55ba5a71c3792e0f29b18f59
Sep 11 03:54:52.246599 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246709 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246709 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246709 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246709 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-web-config\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246924 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246924 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmbl\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-kube-api-access-xvmbl\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246924 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246924 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-out\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.246924 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.246904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.247154 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.247010    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.247154 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.247037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.247154 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.247072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.248438 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.248071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.248438 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.248095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.249909 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:54:52.249893    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle podName:4c23ae5d-ab87-4ba7-959e-aaf78f688e97 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:52.749877001 +0000 UTC m=+67.014355365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97") : configmap references non-existent config key: ca-bundle.crt
Sep 11 03:54:52.250222 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.250198    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.250652 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.250626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.250772 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.250749    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.250946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.250915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.253068 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.253003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.253232 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.253170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-out\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.253852 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.253826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.253928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.253856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-web-config\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.254421 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.254401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.264713 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.264652    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h4xhm"]
Sep 11 03:54:52.267424 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.267401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmbl\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-kube-api-access-xvmbl\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.275640 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:52.275615    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11211d0_1d68_4ab1_9664_a997faf51785.slice/crio-cf4a8aaa4a88c00948ee0a35b6500de565ae0753b1228588caed2ba019643497 WatchSource:0}: Error finding container cf4a8aaa4a88c00948ee0a35b6500de565ae0753b1228588caed2ba019643497: Status 404 returned error can't find the container with id cf4a8aaa4a88c00948ee0a35b6500de565ae0753b1228588caed2ba019643497
Sep 11 03:54:52.291513 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.291489    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sntrc"]
Sep 11 03:54:52.292133 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.292113    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j6dl8\""
Sep 11 03:54:52.294829 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:52.294802    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0800ad_d4e6_4932_86b4_f3e36ba2e7d0.slice/crio-cb81a61a10448da29df233c5cf5ebef5a7518eccc37d6c332764abb7fe5e8ec0 WatchSource:0}: Error finding container cb81a61a10448da29df233c5cf5ebef5a7518eccc37d6c332764abb7fe5e8ec0: Status 404 returned error can't find the container with id cb81a61a10448da29df233c5cf5ebef5a7518eccc37d6c332764abb7fe5e8ec0
Sep 11 03:54:52.300897 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.300863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:52.591173 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.591141    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jt69z"]
Sep 11 03:54:52.594461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.594429    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h4xhm" event={"ID":"e11211d0-1d68-4ab1-9664-a997faf51785","Type":"ContainerStarted","Data":"cf4a8aaa4a88c00948ee0a35b6500de565ae0753b1228588caed2ba019643497"}
Sep 11 03:54:52.596607 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.596581    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64" event={"ID":"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf","Type":"ContainerStarted","Data":"c84ae5b7c1fc261a3788975c66be4897082957209ef7f9d91a3e11ae9b6b8f42"}
Sep 11 03:54:52.596710 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.596640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64" event={"ID":"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf","Type":"ContainerStarted","Data":"d086458a69ae8e3fee0f32a2cde71d1d855205bfb56c6b3eee4ade2711463c5e"}
Sep 11 03:54:52.596710 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.596673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64" event={"ID":"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf","Type":"ContainerStarted","Data":"f33d645c48516cce49e09790093186cd32382b65b1bdd48df41d3517a8bbd70f"}
Sep 11 03:54:52.597868 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.597848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sntrc" event={"ID":"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0","Type":"ContainerStarted","Data":"cb81a61a10448da29df233c5cf5ebef5a7518eccc37d6c332764abb7fe5e8ec0"}
Sep 11 03:54:52.599190 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.599160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj" event={"ID":"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8","Type":"ContainerStarted","Data":"c47aadaf6e76c9f185b35f92c72acfd314466a6a55ba5a71c3792e0f29b18f59"}
Sep 11 03:54:52.605244 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.605173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b47b569c-4qrkh" event={"ID":"a7a3c269-b646-464a-a4a4-2780be3c3c31","Type":"ContainerStarted","Data":"0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415"}
Sep 11 03:54:52.605345 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.605246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b47b569c-4qrkh" event={"ID":"a7a3c269-b646-464a-a4a4-2780be3c3c31","Type":"ContainerStarted","Data":"f87ceb53e59badc48aeaeb756d29eb424edc05578cbbe72b7f2e47d8feaa281d"}
Sep 11 03:54:52.606634 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.606612    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49wg9" event={"ID":"66968c6b-0d96-4040-b342-1de06758e994","Type":"ContainerStarted","Data":"25da1b76f4551539b5cc181e3adf7b4b33273674180092c5b0aed007117e9824"}
Sep 11 03:54:52.624643 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.624586    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67b47b569c-4qrkh" podStartSLOduration=1.62456739 podStartE2EDuration="1.62456739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:52.623829284 +0000 UTC m=+66.888307660" watchObservedRunningTime="2026-09-11 03:54:52.62456739 +0000 UTC m=+66.889045758"
Sep 11 03:54:52.750930 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:52.750897    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023d80c5_5df2_45ce_adaf_a7976e1eef03.slice/crio-5e1b922b4c8bd22c7b25ae0ad9b7e43988ef20f0ea73c6a0155f74100b5aed72 WatchSource:0}: Error finding container 5e1b922b4c8bd22c7b25ae0ad9b7e43988ef20f0ea73c6a0155f74100b5aed72: Status 404 returned error can't find the container with id 5e1b922b4c8bd22c7b25ae0ad9b7e43988ef20f0ea73c6a0155f74100b5aed72
Sep 11 03:54:52.752759 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.752656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.753506 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.753481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:52.956010 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:52.955981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:54:53.263723 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.263669    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:54:53.266090 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:53.266059    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c23ae5d_ab87_4ba7_959e_aaf78f688e97.slice/crio-4d387804c683d2310aa8ec49aa0177b04ebb1dbd98c7ffa17b36e1eadad4c0aa WatchSource:0}: Error finding container 4d387804c683d2310aa8ec49aa0177b04ebb1dbd98c7ffa17b36e1eadad4c0aa: Status 404 returned error can't find the container with id 4d387804c683d2310aa8ec49aa0177b04ebb1dbd98c7ffa17b36e1eadad4c0aa
Sep 11 03:54:53.614581 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.614479    2582 generic.go:356] "Generic (PLEG): container finished" podID="66968c6b-0d96-4040-b342-1de06758e994" containerID="1262100f5d2ac65ca76f2b67200ab648db609e04ae59b86ae641225ebaaca624" exitCode=0
Sep 11 03:54:53.614741 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.614593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49wg9" event={"ID":"66968c6b-0d96-4040-b342-1de06758e994","Type":"ContainerDied","Data":"1262100f5d2ac65ca76f2b67200ab648db609e04ae59b86ae641225ebaaca624"}
Sep 11 03:54:53.617074 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.617048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerStarted","Data":"4d387804c683d2310aa8ec49aa0177b04ebb1dbd98c7ffa17b36e1eadad4c0aa"}
Sep 11 03:54:53.618859 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.618805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jt69z" event={"ID":"023d80c5-5df2-45ce-adaf-a7976e1eef03","Type":"ContainerStarted","Data":"5e1b922b4c8bd22c7b25ae0ad9b7e43988ef20f0ea73c6a0155f74100b5aed72"}
Sep 11 03:54:53.951119 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.951082    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5479598577-fz8wj"]
Sep 11 03:54:53.955298 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.955242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:53.958639 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.958589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 03:54:53.958774 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.958744    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2hvf26vkrl9lv\""
Sep 11 03:54:53.958837 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.958803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pgxjj\""
Sep 11 03:54:53.958889 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.958589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 03:54:53.958999 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.958981    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 03:54:53.959111 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.959077    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 03:54:53.959205 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.959188    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 03:54:53.969341 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:53.969244    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5479598577-fz8wj"]
Sep 11 03:54:54.064805 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.064774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.064970 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.064819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.064970 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.064909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.065136 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.064982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd56357-d39e-47bd-8ddd-d716bb3064da-metrics-client-ca\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.065136 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.065026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.065136 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.065051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-grpc-tls\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.065136 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.065076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xdv\" (UniqueName: \"kubernetes.io/projected/1cd56357-d39e-47bd-8ddd-d716bb3064da-kube-api-access-b6xdv\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.065299 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.065179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-tls\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.166647 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.166855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-grpc-tls\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.166855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xdv\" (UniqueName: \"kubernetes.io/projected/1cd56357-d39e-47bd-8ddd-d716bb3064da-kube-api-access-b6xdv\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.166855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-tls\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.166855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.166855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.167136 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.167136 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.166905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd56357-d39e-47bd-8ddd-d716bb3064da-metrics-client-ca\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.167623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.167595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd56357-d39e-47bd-8ddd-d716bb3064da-metrics-client-ca\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.172288 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.172181    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.172578 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.172498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-grpc-tls\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.172685 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.172639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.174004 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.173946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.174174 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.174111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.174288 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.174247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1cd56357-d39e-47bd-8ddd-d716bb3064da-secret-thanos-querier-tls\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.177095 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.176994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xdv\" (UniqueName: \"kubernetes.io/projected/1cd56357-d39e-47bd-8ddd-d716bb3064da-kube-api-access-b6xdv\") pod \"thanos-querier-5479598577-fz8wj\" (UID: \"1cd56357-d39e-47bd-8ddd-d716bb3064da\") " pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.270801 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.270721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:54:54.630305 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.630184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49wg9" event={"ID":"66968c6b-0d96-4040-b342-1de06758e994","Type":"ContainerStarted","Data":"6bedae876f7861b0ff9baebff018da29edbb627bbdb11dad55245f7124f52ee5"}
Sep 11 03:54:54.639473 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.638905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64" event={"ID":"7164ec5f-d1cf-41e7-84f1-28405bdaaaaf","Type":"ContainerStarted","Data":"3d7fcca5ae2c5adc999cadfbc067add8750416b48e86ac36682decde56337685"}
Sep 11 03:54:54.645191 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.645163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj" event={"ID":"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8","Type":"ContainerStarted","Data":"df9a2c7d31692795a01183f1dda889c0f3d5c09f6a257bc1e16f2c3a23c89cec"}
Sep 11 03:54:54.660748 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.660623    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-bsg64" podStartSLOduration=2.482951194 podStartE2EDuration="4.660600992s" totalImagesPullingTime="2.177649798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:54.659753115 +0000 UTC m=+68.924231487" watchObservedRunningTime="2026-09-11 03:54:54.660600992 +0000 UTC m=+68.925079367"
Sep 11 03:54:54.798283 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:54.798183    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5479598577-fz8wj"]
Sep 11 03:54:55.119120 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:55.119082    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd56357_d39e_47bd_8ddd_d716bb3064da.slice/crio-d35da5bee6e981473b2ed67d2e2409f72cbf80ca9d3160ed3936f5362f2acbe6 WatchSource:0}: Error finding container d35da5bee6e981473b2ed67d2e2409f72cbf80ca9d3160ed3936f5362f2acbe6: Status 404 returned error can't find the container with id d35da5bee6e981473b2ed67d2e2409f72cbf80ca9d3160ed3936f5362f2acbe6
Sep 11 03:54:55.371736 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.371646    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6777697447-sl8dp"]
Sep 11 03:54:55.375367 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.375312    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.378616 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.378589    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 03:54:55.378725 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.378626    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 03:54:55.379413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.379394    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5hkdfrk8g5c53\""
Sep 11 03:54:55.380347 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.379424    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dxwrh\""
Sep 11 03:54:55.380347 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.379482    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 03:54:55.380347 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.379506    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 03:54:55.385365 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.385305    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6777697447-sl8dp"]
Sep 11 03:54:55.482044 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.482004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4str\" (UniqueName: \"kubernetes.io/projected/9a24aaa1-764e-43ee-a16c-21022348b7bf-kube-api-access-g4str\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.482223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.482094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-secret-metrics-server-client-certs\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.482223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.482119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-client-ca-bundle\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.482223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.482140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a24aaa1-764e-43ee-a16c-21022348b7bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.482223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.482188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a24aaa1-764e-43ee-a16c-21022348b7bf-metrics-server-audit-profiles\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.482439 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.482232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-secret-metrics-server-tls\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.482439 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.482267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a24aaa1-764e-43ee-a16c-21022348b7bf-audit-log\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.583718 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.583665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-secret-metrics-server-client-certs\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.583718 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.583718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-client-ca-bundle\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.583946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.583749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a24aaa1-764e-43ee-a16c-21022348b7bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.583946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.583803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a24aaa1-764e-43ee-a16c-21022348b7bf-metrics-server-audit-profiles\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.583946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.583836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-secret-metrics-server-tls\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.583946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.583871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a24aaa1-764e-43ee-a16c-21022348b7bf-audit-log\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.583946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.583898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4str\" (UniqueName: \"kubernetes.io/projected/9a24aaa1-764e-43ee-a16c-21022348b7bf-kube-api-access-g4str\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.584827 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.584798    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a24aaa1-764e-43ee-a16c-21022348b7bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.584955 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.584923    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a24aaa1-764e-43ee-a16c-21022348b7bf-audit-log\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.585037 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.585008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a24aaa1-764e-43ee-a16c-21022348b7bf-metrics-server-audit-profiles\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.586939 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.586906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-secret-metrics-server-client-certs\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.587180 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.587135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-client-ca-bundle\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.587405 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.587382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a24aaa1-764e-43ee-a16c-21022348b7bf-secret-metrics-server-tls\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.593058 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.593036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4str\" (UniqueName: \"kubernetes.io/projected/9a24aaa1-764e-43ee-a16c-21022348b7bf-kube-api-access-g4str\") pod \"metrics-server-6777697447-sl8dp\" (UID: \"9a24aaa1-764e-43ee-a16c-21022348b7bf\") " pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.652476 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.652389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-49wg9" event={"ID":"66968c6b-0d96-4040-b342-1de06758e994","Type":"ContainerStarted","Data":"936a91e10ed93ed8c7d2b73e0dafce67d7e3541a9b78d0e720d08ff51031b1c3"}
Sep 11 03:54:55.654475 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.654448    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerID="333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3" exitCode=0
Sep 11 03:54:55.654621 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.654533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3"}
Sep 11 03:54:55.656717 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.656681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sntrc" event={"ID":"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0","Type":"ContainerStarted","Data":"f41c90da160d4d3108e8642722e04eabd2f36d023e263174ac642b9bf2c256a7"}
Sep 11 03:54:55.656858 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.656732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sntrc" event={"ID":"4a0800ad-d4e6-4932-86b4-f3e36ba2e7d0","Type":"ContainerStarted","Data":"cf86d6e805b8ecd9f9e8abcc06824633d727be8785389ada1b8d7afde3300846"}
Sep 11 03:54:55.659241 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.659216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj" event={"ID":"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8","Type":"ContainerStarted","Data":"8d5aea49d0e9bb14bcb21b68f7d090ff373c441d2e7ffa913b325f653282099c"}
Sep 11 03:54:55.659322 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.659270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj" event={"ID":"0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8","Type":"ContainerStarted","Data":"0383bef864d72a4e2a8aecb6b10fe139d472c0d9bc3c32b1f4eec4f7d683b753"}
Sep 11 03:54:55.660480 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.660456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" event={"ID":"1cd56357-d39e-47bd-8ddd-d716bb3064da","Type":"ContainerStarted","Data":"d35da5bee6e981473b2ed67d2e2409f72cbf80ca9d3160ed3936f5362f2acbe6"}
Sep 11 03:54:55.676171 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.676122    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-49wg9" podStartSLOduration=4.75356017 podStartE2EDuration="5.676101521s" totalImagesPullingTime="846.422271ms" totalInitContainerRuntime="76.11908ms" isStatefulPod=true podCreationTimestamp="2026-09-11 03:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:55.674795528 +0000 UTC m=+69.939273899" watchObservedRunningTime="2026-09-11 03:54:55.676101521 +0000 UTC m=+69.940579892"
Sep 11 03:54:55.689357 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.689332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:54:55.760227 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.760175    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sntrc" podStartSLOduration=67.557190858 podStartE2EDuration="1m9.760156702s" totalImagesPullingTime="2.202965844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:55.758923859 +0000 UTC m=+70.023402230" watchObservedRunningTime="2026-09-11 03:54:55.760156702 +0000 UTC m=+70.024635072"
Sep 11 03:54:55.816319 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.816254    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zbjgj" podStartSLOduration=3.536239257 podStartE2EDuration="5.816236074s" totalImagesPullingTime="2.279996817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:55.815045959 +0000 UTC m=+70.079524330" watchObservedRunningTime="2026-09-11 03:54:55.816236074 +0000 UTC m=+70.080714444"
Sep 11 03:54:55.838599 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.838331    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"]
Sep 11 03:54:55.843191 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.842708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:55.846532 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.846503    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 03:54:55.846532 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.846520    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rl69c\""
Sep 11 03:54:55.853641 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.853473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"]
Sep 11 03:54:55.887487 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.887447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ztt\" (UniqueName: \"kubernetes.io/projected/967439c6-58d9-41ae-8ad3-74def7677aa7-kube-api-access-l9ztt\") pod \"monitoring-plugin-5d85778b9d-2pk48\" (UID: \"967439c6-58d9-41ae-8ad3-74def7677aa7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:55.887663 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.887517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967439c6-58d9-41ae-8ad3-74def7677aa7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2pk48\" (UID: \"967439c6-58d9-41ae-8ad3-74def7677aa7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:55.936381 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.936331    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:55.936576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.936413    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:55.942463 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.942434    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:55.988375 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.988309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ztt\" (UniqueName: \"kubernetes.io/projected/967439c6-58d9-41ae-8ad3-74def7677aa7-kube-api-access-l9ztt\") pod \"monitoring-plugin-5d85778b9d-2pk48\" (UID: \"967439c6-58d9-41ae-8ad3-74def7677aa7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:55.988546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.988415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967439c6-58d9-41ae-8ad3-74def7677aa7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2pk48\" (UID: \"967439c6-58d9-41ae-8ad3-74def7677aa7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:55.991047 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.991025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/967439c6-58d9-41ae-8ad3-74def7677aa7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2pk48\" (UID: \"967439c6-58d9-41ae-8ad3-74def7677aa7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:55.996458 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:55.996432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ztt\" (UniqueName: \"kubernetes.io/projected/967439c6-58d9-41ae-8ad3-74def7677aa7-kube-api-access-l9ztt\") pod \"monitoring-plugin-5d85778b9d-2pk48\" (UID: \"967439c6-58d9-41ae-8ad3-74def7677aa7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:56.156699 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.156662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:54:56.670462 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.670400    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:54:56.782111 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.782078    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b47b569c-4qrkh"]
Sep 11 03:54:56.816808 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.816585    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f87949f6-pnc8k"]
Sep 11 03:54:56.820048 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.820025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.828262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.828035    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f87949f6-pnc8k"]
Sep 11 03:54:56.897828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.897689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnpw\" (UniqueName: \"kubernetes.io/projected/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-kube-api-access-vbnpw\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.897828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.897799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-serving-cert\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.898071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.897902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-oauth-config\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.898071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.897968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-config\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.898071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.898005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-oauth-serving-cert\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.898071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.898029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-service-ca\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.898071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.898044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-trusted-ca-bundle\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.999631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.999027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-service-ca\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.999631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.999075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-trusted-ca-bundle\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.999631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.999105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnpw\" (UniqueName: \"kubernetes.io/projected/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-kube-api-access-vbnpw\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.999631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.999178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-serving-cert\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.999631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.999220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-oauth-config\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.999631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.999261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-config\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:56.999631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:56.999318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-oauth-serving-cert\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.000844 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.000159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-trusted-ca-bundle\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.000844 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.000770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-service-ca\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.001036 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.000994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-oauth-serving-cert\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.001306 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.001264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-config\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.003135 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.003108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-serving-cert\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.003671 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.003648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-oauth-config\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.010062 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.009413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnpw\" (UniqueName: \"kubernetes.io/projected/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-kube-api-access-vbnpw\") pod \"console-66f87949f6-pnc8k\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") " pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.132630 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.132358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:54:57.225600 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.224861    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:54:57.232690 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.231291    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.235262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.234272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 03:54:57.235262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.234638    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 03:54:57.235262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.234875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sq7dp\""
Sep 11 03:54:57.235262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.234939    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 03:54:57.235262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.235125    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 03:54:57.236944 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.236054    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 03:54:57.236944 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.236192    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 03:54:57.236944 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.236394    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 03:54:57.236944 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.236645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 03:54:57.236944 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.236897    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 03:54:57.237282 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.237191    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 03:54:57.238489 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.237413    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 03:54:57.238489 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.237619    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ajhbijbpkpmo\""
Sep 11 03:54:57.239103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.238944    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 03:54:57.239467 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.239305    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 03:54:57.240836 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.240812    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 03:54:57.246549 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.246524    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:54:57.302849 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.302704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.302849 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.302838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.302888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.302950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303177    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnts\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-kube-api-access-rvnts\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.304320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.303674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.306225 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.306046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.306225 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.306083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407517 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407714 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407714 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407714 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407863 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407863 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407863 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407863 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.407863 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnts\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-kube-api-access-rvnts\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.407983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.408017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.408042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.408087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.408116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.408148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.408237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.408166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.409043 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.408699    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.409423 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.409216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.410828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.410804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.411124 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.411101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.411482 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.411460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.411613 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.411495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.411905 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.411877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.412071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.412018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.412991 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.412949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.413627 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.413581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.414372 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.414323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.414489 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.414406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.415075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.414621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.415328 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.415284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.416020 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.415989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.416497 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.416451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.416497 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.416462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.416840 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.416816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.416914 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.416818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.418714 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.418690    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnts\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-kube-api-access-rvnts\") pod \"prometheus-k8s-0\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:57.549508 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:57.549178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:54:58.566905 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:58.566825    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"]
Sep 11 03:54:58.897476 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:58.897383    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967439c6_58d9_41ae_8ad3_74def7677aa7.slice/crio-a1ede1df8b8d013b5962995492ae2cf55088132a1df7bd4b6170b1bcd17bd449 WatchSource:0}: Error finding container a1ede1df8b8d013b5962995492ae2cf55088132a1df7bd4b6170b1bcd17bd449: Status 404 returned error can't find the container with id a1ede1df8b8d013b5962995492ae2cf55088132a1df7bd4b6170b1bcd17bd449
Sep 11 03:54:59.191356 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.191279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f87949f6-pnc8k"]
Sep 11 03:54:59.195201 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.195156    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6777697447-sl8dp"]
Sep 11 03:54:59.202653 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.202622    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:54:59.406929 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:59.406731    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a24aaa1_764e_43ee_a16c_21022348b7bf.slice/crio-670bd61459bd8b90cf9734059a5e83d2455c53c5b31cc0ad7ff8b2a98bdb6aa7 WatchSource:0}: Error finding container 670bd61459bd8b90cf9734059a5e83d2455c53c5b31cc0ad7ff8b2a98bdb6aa7: Status 404 returned error can't find the container with id 670bd61459bd8b90cf9734059a5e83d2455c53c5b31cc0ad7ff8b2a98bdb6aa7
Sep 11 03:54:59.407383 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:59.407357    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf59fa8_d04b_45c3_8fce_364fa08bcad3.slice/crio-92411f2aed6e5b0aeb7fb4c277400e5bee6f0390bd7c0afbaa071f3af409b72f WatchSource:0}: Error finding container 92411f2aed6e5b0aeb7fb4c277400e5bee6f0390bd7c0afbaa071f3af409b72f: Status 404 returned error can't find the container with id 92411f2aed6e5b0aeb7fb4c277400e5bee6f0390bd7c0afbaa071f3af409b72f
Sep 11 03:54:59.408463 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:54:59.408443    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ea8b90_1bf1_4a3a_bc8c_ae6cd506ad8b.slice/crio-057ae9be66d0749347cf1cca8f7595f3aeacb15c5aca8d13aeb8b924c31e6bd8 WatchSource:0}: Error finding container 057ae9be66d0749347cf1cca8f7595f3aeacb15c5aca8d13aeb8b924c31e6bd8: Status 404 returned error can't find the container with id 057ae9be66d0749347cf1cca8f7595f3aeacb15c5aca8d13aeb8b924c31e6bd8
Sep 11 03:54:59.681853 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.681821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" event={"ID":"1cd56357-d39e-47bd-8ddd-d716bb3064da","Type":"ContainerStarted","Data":"fce8c864c55065bc970190c7875654e29c915ac58ed686bda16b605fb09309c6"}
Sep 11 03:54:59.682283 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.681948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" event={"ID":"1cd56357-d39e-47bd-8ddd-d716bb3064da","Type":"ContainerStarted","Data":"c0e275bcbb6485a614881a31137d26e125e79ab506dd9175dd4128062091c20c"}
Sep 11 03:54:59.682283 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.681982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" event={"ID":"1cd56357-d39e-47bd-8ddd-d716bb3064da","Type":"ContainerStarted","Data":"5346b86d82e0ac5eb0721dd24510dc1366f070c7617cb917fe96f95411fb0c93"}
Sep 11 03:54:59.686436 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.686394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerStarted","Data":"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"}
Sep 11 03:54:59.688952 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.688808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f87949f6-pnc8k" event={"ID":"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b","Type":"ContainerStarted","Data":"591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739"}
Sep 11 03:54:59.688952 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.688865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f87949f6-pnc8k" event={"ID":"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b","Type":"ContainerStarted","Data":"057ae9be66d0749347cf1cca8f7595f3aeacb15c5aca8d13aeb8b924c31e6bd8"}
Sep 11 03:54:59.691023 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.690995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jt69z" event={"ID":"023d80c5-5df2-45ce-adaf-a7976e1eef03","Type":"ContainerStarted","Data":"519ea0bfa836e43bf644cccaf8863494e4cb5b0599d450fca301adad10755871"}
Sep 11 03:54:59.691107 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.691091    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:54:59.693184 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.693158    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerID="fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a" exitCode=0
Sep 11 03:54:59.693270 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.693190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a"}
Sep 11 03:54:59.693270 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.693253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerStarted","Data":"92411f2aed6e5b0aeb7fb4c277400e5bee6f0390bd7c0afbaa071f3af409b72f"}
Sep 11 03:54:59.694768 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.694744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6777697447-sl8dp" event={"ID":"9a24aaa1-764e-43ee-a16c-21022348b7bf","Type":"ContainerStarted","Data":"670bd61459bd8b90cf9734059a5e83d2455c53c5b31cc0ad7ff8b2a98bdb6aa7"}
Sep 11 03:54:59.697136 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.697092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48" event={"ID":"967439c6-58d9-41ae-8ad3-74def7677aa7","Type":"ContainerStarted","Data":"a1ede1df8b8d013b5962995492ae2cf55088132a1df7bd4b6170b1bcd17bd449"}
Sep 11 03:54:59.699420 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.699023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h4xhm" event={"ID":"e11211d0-1d68-4ab1-9664-a997faf51785","Type":"ContainerStarted","Data":"07d42a6d92ff7451d90c0f5a7bd9a00942f54ba4a52ce1f083539b589ebc96c1"}
Sep 11 03:54:59.712690 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.711790    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66f87949f6-pnc8k" podStartSLOduration=3.711771947 podStartE2EDuration="3.711771947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:59.711770792 +0000 UTC m=+73.976249163" watchObservedRunningTime="2026-09-11 03:54:59.711771947 +0000 UTC m=+73.976250319"
Sep 11 03:54:59.730664 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.730597    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h4xhm" podStartSLOduration=67.107435303 podStartE2EDuration="1m13.730584787s" totalImagesPullingTime="6.623149484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:59.730262651 +0000 UTC m=+73.994741023" watchObservedRunningTime="2026-09-11 03:54:59.730584787 +0000 UTC m=+73.995063159"
Sep 11 03:54:59.773546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:54:59.773481    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jt69z" podStartSLOduration=67.624815056 podStartE2EDuration="1m13.77345854s" totalImagesPullingTime="6.148643484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:59.772351167 +0000 UTC m=+74.036829531" watchObservedRunningTime="2026-09-11 03:54:59.77345854 +0000 UTC m=+74.037936992"
Sep 11 03:55:00.706834 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:00.706785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerStarted","Data":"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"}
Sep 11 03:55:00.707327 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:00.706911    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerStarted","Data":"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"}
Sep 11 03:55:00.707327 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:00.706970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerStarted","Data":"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"}
Sep 11 03:55:00.707327 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:00.707023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerStarted","Data":"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"}
Sep 11 03:55:01.716327 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.716292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6777697447-sl8dp" event={"ID":"9a24aaa1-764e-43ee-a16c-21022348b7bf","Type":"ContainerStarted","Data":"ba5b095a041f84533a20a30309848ef12ca485ce989a11d2f5c853c5b01e276f"}
Sep 11 03:55:01.718361 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.718332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48" event={"ID":"967439c6-58d9-41ae-8ad3-74def7677aa7","Type":"ContainerStarted","Data":"10187bd7a9bae76069dfb457d61c3d06d9502f6ab36ac4f52374c5ac13cdf8f2"}
Sep 11 03:55:01.718535 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.718516    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:55:01.722571 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.722536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" event={"ID":"1cd56357-d39e-47bd-8ddd-d716bb3064da","Type":"ContainerStarted","Data":"4a5c8d581670047e53975fdd10f3683a6c847ddff7454854e0ebbe3202ef773c"}
Sep 11 03:55:01.722674 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.722626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" event={"ID":"1cd56357-d39e-47bd-8ddd-d716bb3064da","Type":"ContainerStarted","Data":"fdd51ff18feb918cfdccc33508e30bebece868a41bccfb140215c6815e85f373"}
Sep 11 03:55:01.722738 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.722686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" event={"ID":"1cd56357-d39e-47bd-8ddd-d716bb3064da","Type":"ContainerStarted","Data":"e9d969a2fa32765596e8fca46880909a610f0d217593752c36b9d76a54c556ef"}
Sep 11 03:55:01.722790 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.722757    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:55:01.724971 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.724936    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48"
Sep 11 03:55:01.727382 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.727364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerStarted","Data":"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"}
Sep 11 03:55:01.738827 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.738779    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6777697447-sl8dp" podStartSLOduration=4.949468899 podStartE2EDuration="6.738762754s" totalImagesPullingTime="1.789293855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:01.737690882 +0000 UTC m=+76.002169254" watchObservedRunningTime="2026-09-11 03:55:01.738762754 +0000 UTC m=+76.003241127"
Sep 11 03:55:01.767047 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.766990    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.7146495 podStartE2EDuration="9.766970815s" totalImagesPullingTime="7.032111651s" totalInitContainerRuntime="20.209664ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:01.766710586 +0000 UTC m=+76.031188960" watchObservedRunningTime="2026-09-11 03:55:01.766970815 +0000 UTC m=+76.031449202"
Sep 11 03:55:01.785856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.785797    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pk48" podStartSLOduration=4.494768221 podStartE2EDuration="6.785775731s" totalImagesPullingTime="2.29100751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:01.785724483 +0000 UTC m=+76.050202866" watchObservedRunningTime="2026-09-11 03:55:01.785775731 +0000 UTC m=+76.050254103"
Sep 11 03:55:01.811884 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.811831    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj" podStartSLOduration=3.387798929 podStartE2EDuration="8.811815425s" totalImagesPullingTime="5.424016496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:01.809006896 +0000 UTC m=+76.073485265" watchObservedRunningTime="2026-09-11 03:55:01.811815425 +0000 UTC m=+76.076293797"
Sep 11 03:55:01.887931 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:01.887886    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:55:03.740889 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:03.740844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerStarted","Data":"d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1"}
Sep 11 03:55:03.741272 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:03.740955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerStarted","Data":"a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436"}
Sep 11 03:55:03.741272 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:03.740983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerStarted","Data":"5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e"}
Sep 11 03:55:03.741272 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:03.741014    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerStarted","Data":"c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a"}
Sep 11 03:55:03.741272 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:03.741048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerStarted","Data":"399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226"}
Sep 11 03:55:03.741272 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:03.741072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerStarted","Data":"2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e"}
Sep 11 03:55:03.770665 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:03.770609    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.417091892 podStartE2EDuration="6.770594052s" totalImagesPullingTime="3.328184598s" totalInitContainerRuntime="25.317562ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:03.767925441 +0000 UTC m=+78.032403809" watchObservedRunningTime="2026-09-11 03:55:03.770594052 +0000 UTC m=+78.035072423"
Sep 11 03:55:07.133117 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:07.133079    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:55:07.133501 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:07.133162    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:55:07.137859 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:07.137836    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:55:07.550152 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:07.550116    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:55:07.739403 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:07.739374    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5479598577-fz8wj"
Sep 11 03:55:07.762173 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:07.762147    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:55:07.829279 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:07.829186    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f744bc99-n8vh8"]
Sep 11 03:55:15.689815 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:15.689766    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:55:15.690317 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:15.689848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:55:21.808164 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:21.808119    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67b47b569c-4qrkh" podUID="a7a3c269-b646-464a-a4a4-2780be3c3c31" containerName="console" containerID="cri-o://0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415" gracePeriod=15
Sep 11 03:55:22.044338 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.044315    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:55:22.140874 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.140784    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-service-ca\") pod \"a7a3c269-b646-464a-a4a4-2780be3c3c31\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") "
Sep 11 03:55:22.140874 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.140870    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-config\") pod \"a7a3c269-b646-464a-a4a4-2780be3c3c31\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") "
Sep 11 03:55:22.141092 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.140907    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-trusted-ca-bundle\") pod \"a7a3c269-b646-464a-a4a4-2780be3c3c31\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") "
Sep 11 03:55:22.141092 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.140944    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lt8g\" (UniqueName: \"kubernetes.io/projected/a7a3c269-b646-464a-a4a4-2780be3c3c31-kube-api-access-2lt8g\") pod \"a7a3c269-b646-464a-a4a4-2780be3c3c31\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") "
Sep 11 03:55:22.141092 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141011    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-serving-cert\") pod \"a7a3c269-b646-464a-a4a4-2780be3c3c31\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") "
Sep 11 03:55:22.141304 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-oauth-serving-cert\") pod \"a7a3c269-b646-464a-a4a4-2780be3c3c31\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") "
Sep 11 03:55:22.141304 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-oauth-config\") pod \"a7a3c269-b646-464a-a4a4-2780be3c3c31\" (UID: \"a7a3c269-b646-464a-a4a4-2780be3c3c31\") "
Sep 11 03:55:22.141304 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141151    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7a3c269-b646-464a-a4a4-2780be3c3c31" (UID: "a7a3c269-b646-464a-a4a4-2780be3c3c31"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:22.141304 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141277    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-config" (OuterVolumeSpecName: "console-config") pod "a7a3c269-b646-464a-a4a4-2780be3c3c31" (UID: "a7a3c269-b646-464a-a4a4-2780be3c3c31"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:22.141484 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141304    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a7a3c269-b646-464a-a4a4-2780be3c3c31" (UID: "a7a3c269-b646-464a-a4a4-2780be3c3c31"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:22.141484 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141420    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:22.141581 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.141511    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7a3c269-b646-464a-a4a4-2780be3c3c31" (UID: "a7a3c269-b646-464a-a4a4-2780be3c3c31"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:22.143401 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.143365    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7a3c269-b646-464a-a4a4-2780be3c3c31" (UID: "a7a3c269-b646-464a-a4a4-2780be3c3c31"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:22.143503 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.143461    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a3c269-b646-464a-a4a4-2780be3c3c31-kube-api-access-2lt8g" (OuterVolumeSpecName: "kube-api-access-2lt8g") pod "a7a3c269-b646-464a-a4a4-2780be3c3c31" (UID: "a7a3c269-b646-464a-a4a4-2780be3c3c31"). InnerVolumeSpecName "kube-api-access-2lt8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:22.143503 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.143458    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7a3c269-b646-464a-a4a4-2780be3c3c31" (UID: "a7a3c269-b646-464a-a4a4-2780be3c3c31"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:22.242584 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.242509    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-oauth-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:22.242584 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.242543    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-oauth-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:22.242584 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.242583    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:22.242584 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.242592    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7a3c269-b646-464a-a4a4-2780be3c3c31-trusted-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:22.242845 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.242603    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lt8g\" (UniqueName: \"kubernetes.io/projected/a7a3c269-b646-464a-a4a4-2780be3c3c31-kube-api-access-2lt8g\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:22.242845 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.242613    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a3c269-b646-464a-a4a4-2780be3c3c31-console-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:22.820946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.820905    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7a3c269-b646-464a-a4a4-2780be3c3c31" containerID="0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415" exitCode=0
Sep 11 03:55:22.821350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.820974    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b47b569c-4qrkh"
Sep 11 03:55:22.821350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.820984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b47b569c-4qrkh" event={"ID":"a7a3c269-b646-464a-a4a4-2780be3c3c31","Type":"ContainerDied","Data":"0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415"}
Sep 11 03:55:22.821350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.821035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b47b569c-4qrkh" event={"ID":"a7a3c269-b646-464a-a4a4-2780be3c3c31","Type":"ContainerDied","Data":"f87ceb53e59badc48aeaeb756d29eb424edc05578cbbe72b7f2e47d8feaa281d"}
Sep 11 03:55:22.821350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.821055    2582 scope.go:117] "RemoveContainer" containerID="0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415"
Sep 11 03:55:22.830045 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.830030    2582 scope.go:117] "RemoveContainer" containerID="0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415"
Sep 11 03:55:22.830309 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:55:22.830289    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415\": container with ID starting with 0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415 not found: ID does not exist" containerID="0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415"
Sep 11 03:55:22.830382 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.830321    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415"} err="failed to get container status \"0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415\": rpc error: code = NotFound desc = could not find container \"0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415\": container with ID starting with 0a001f4b3cba466c6b1182be771a19fe463c051bb57d7cc48db7317edfc19415 not found: ID does not exist"
Sep 11 03:55:22.837784 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.837760    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b47b569c-4qrkh"]
Sep 11 03:55:22.841324 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:22.841302    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67b47b569c-4qrkh"]
Sep 11 03:55:24.273943 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:24.273904    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a3c269-b646-464a-a4a4-2780be3c3c31" path="/var/lib/kubelet/pods/a7a3c269-b646-464a-a4a4-2780be3c3c31/volumes"
Sep 11 03:55:30.709464 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:30.709425    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jt69z"
Sep 11 03:55:32.853840 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:32.853798    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84f744bc99-n8vh8" podUID="940fdf27-b346-4804-a8c9-a362daed6fa4" containerName="console" containerID="cri-o://b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624" gracePeriod=15
Sep 11 03:55:33.111723 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.111666    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:55:33.139984 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.139960    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-console-config\") pod \"940fdf27-b346-4804-a8c9-a362daed6fa4\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") "
Sep 11 03:55:33.140121 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.139999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-serving-cert\") pod \"940fdf27-b346-4804-a8c9-a362daed6fa4\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") "
Sep 11 03:55:33.140121 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.140019    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-service-ca\") pod \"940fdf27-b346-4804-a8c9-a362daed6fa4\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") "
Sep 11 03:55:33.140121 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.140076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-oauth-serving-cert\") pod \"940fdf27-b346-4804-a8c9-a362daed6fa4\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") "
Sep 11 03:55:33.140121 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.140103    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9g5v\" (UniqueName: \"kubernetes.io/projected/940fdf27-b346-4804-a8c9-a362daed6fa4-kube-api-access-w9g5v\") pod \"940fdf27-b346-4804-a8c9-a362daed6fa4\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") "
Sep 11 03:55:33.140319 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.140146    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-oauth-config\") pod \"940fdf27-b346-4804-a8c9-a362daed6fa4\" (UID: \"940fdf27-b346-4804-a8c9-a362daed6fa4\") "
Sep 11 03:55:33.140416 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.140389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-console-config" (OuterVolumeSpecName: "console-config") pod "940fdf27-b346-4804-a8c9-a362daed6fa4" (UID: "940fdf27-b346-4804-a8c9-a362daed6fa4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:33.140490 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.140471    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "940fdf27-b346-4804-a8c9-a362daed6fa4" (UID: "940fdf27-b346-4804-a8c9-a362daed6fa4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:33.140542 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.140498    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-service-ca" (OuterVolumeSpecName: "service-ca") pod "940fdf27-b346-4804-a8c9-a362daed6fa4" (UID: "940fdf27-b346-4804-a8c9-a362daed6fa4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:33.142403 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.142356    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940fdf27-b346-4804-a8c9-a362daed6fa4-kube-api-access-w9g5v" (OuterVolumeSpecName: "kube-api-access-w9g5v") pod "940fdf27-b346-4804-a8c9-a362daed6fa4" (UID: "940fdf27-b346-4804-a8c9-a362daed6fa4"). InnerVolumeSpecName "kube-api-access-w9g5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:33.142519 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.142467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "940fdf27-b346-4804-a8c9-a362daed6fa4" (UID: "940fdf27-b346-4804-a8c9-a362daed6fa4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:33.142611 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.142537    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "940fdf27-b346-4804-a8c9-a362daed6fa4" (UID: "940fdf27-b346-4804-a8c9-a362daed6fa4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:33.241570 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.241520    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-console-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:33.241570 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.241548    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:33.241570 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.241576    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:33.241855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.241585    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940fdf27-b346-4804-a8c9-a362daed6fa4-oauth-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:33.241855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.241593    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9g5v\" (UniqueName: \"kubernetes.io/projected/940fdf27-b346-4804-a8c9-a362daed6fa4-kube-api-access-w9g5v\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:33.241855 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.241601    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940fdf27-b346-4804-a8c9-a362daed6fa4-console-oauth-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:55:33.868872 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.868836    2582 generic.go:356] "Generic (PLEG): container finished" podID="940fdf27-b346-4804-a8c9-a362daed6fa4" containerID="b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624" exitCode=0
Sep 11 03:55:33.869249 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.868879    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f744bc99-n8vh8"
Sep 11 03:55:33.869249 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.868923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f744bc99-n8vh8" event={"ID":"940fdf27-b346-4804-a8c9-a362daed6fa4","Type":"ContainerDied","Data":"b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624"}
Sep 11 03:55:33.869249 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.868984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f744bc99-n8vh8" event={"ID":"940fdf27-b346-4804-a8c9-a362daed6fa4","Type":"ContainerDied","Data":"b08c98ac99117c396b2f96e21d3b207d97c212471949c96bdf59a94bcec3a08f"}
Sep 11 03:55:33.869249 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.869006    2582 scope.go:117] "RemoveContainer" containerID="b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624"
Sep 11 03:55:33.879230 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.878693    2582 scope.go:117] "RemoveContainer" containerID="b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624"
Sep 11 03:55:33.881737 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:55:33.881566    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624\": container with ID starting with b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624 not found: ID does not exist" containerID="b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624"
Sep 11 03:55:33.881824 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.881745    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624"} err="failed to get container status \"b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624\": rpc error: code = NotFound desc = could not find container \"b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624\": container with ID starting with b1b30bc834da9df9f9114b166d2cb9e67553f01697b63efc446039104e75d624 not found: ID does not exist"
Sep 11 03:55:33.893460 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.893431    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f744bc99-n8vh8"]
Sep 11 03:55:33.897903 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:33.897880    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84f744bc99-n8vh8"]
Sep 11 03:55:34.274096 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:34.274063    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940fdf27-b346-4804-a8c9-a362daed6fa4" path="/var/lib/kubelet/pods/940fdf27-b346-4804-a8c9-a362daed6fa4/volumes"
Sep 11 03:55:35.695612 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:35.695579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:55:35.699442 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:35.699415    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6777697447-sl8dp"
Sep 11 03:55:39.946478 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:39.946437    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f658l_fba0f58a-3507-49cb-9909-583301b4abfb/serve-healthcheck-canary/0.log"
Sep 11 03:55:57.550410 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:57.550376    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:55:57.569788 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:57.569760    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:55:57.982126 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:55:57.982101    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:11.250059 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:11.250024    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:56:11.250572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:11.250289    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="alertmanager" containerID="cri-o://1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc" gracePeriod=120
Sep 11 03:56:11.250572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:11.250354    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-metric" containerID="cri-o://3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5" gracePeriod=120
Sep 11 03:56:11.250572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:11.250374    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-web" containerID="cri-o://933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7" gracePeriod=120
Sep 11 03:56:11.250572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:11.250409    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="config-reloader" containerID="cri-o://3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8" gracePeriod=120
Sep 11 03:56:11.250572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:11.250454    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="prom-label-proxy" containerID="cri-o://f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a" gracePeriod=120
Sep 11 03:56:11.250572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:11.250454    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy" containerID="cri-o://a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db" gracePeriod=120
Sep 11 03:56:12.023419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023383    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerID="f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a" exitCode=0
Sep 11 03:56:12.023419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023412    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerID="a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db" exitCode=0
Sep 11 03:56:12.023419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023422    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerID="3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8" exitCode=0
Sep 11 03:56:12.023673 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023431    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerID="1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc" exitCode=0
Sep 11 03:56:12.023673 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"}
Sep 11 03:56:12.023673 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"}
Sep 11 03:56:12.023673 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"}
Sep 11 03:56:12.023673 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.023583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"}
Sep 11 03:56:12.487073 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.487051    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:12.579392 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579366    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-out\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579412    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-tls-assets\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-metrics-client-ca\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-web-config\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579532    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-cluster-tls-config\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579569    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-volume\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.579623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-web\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.580061 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579634    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-main-tls\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.580061 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvmbl\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-kube-api-access-xvmbl\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.580061 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579703    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.580061 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579739    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-main-db\") pod \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\" (UID: \"4c23ae5d-ab87-4ba7-959e-aaf78f688e97\") "
Sep 11 03:56:12.580061 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.579975    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:12.580310 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.580210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:12.580367 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.580335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:12.582942 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.582910    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:12.583305 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.583246    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:12.583843 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.583703    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:12.583843 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.583773    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:12.583843 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.583812    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-out" (OuterVolumeSpecName: "config-out") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:12.584109 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.584078    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:12.584452 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.584421    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-kube-api-access-xvmbl" (OuterVolumeSpecName: "kube-api-access-xvmbl") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "kube-api-access-xvmbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:12.584844 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.584824    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-volume" (OuterVolumeSpecName: "config-volume") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:12.589052 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.589002    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:12.595185 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.595165    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-web-config" (OuterVolumeSpecName: "web-config") pod "4c23ae5d-ab87-4ba7-959e-aaf78f688e97" (UID: "4c23ae5d-ab87-4ba7-959e-aaf78f688e97"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:12.680460 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680430    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-main-tls\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680460 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680454    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmbl\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-kube-api-access-xvmbl\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680460 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680465    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680474    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-main-db\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680483    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-out\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680492    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-tls-assets\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680501    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680509    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-metrics-client-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680517    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-web-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680525    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680533    2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-cluster-tls-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680542    2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:12.680646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:12.680550    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c23ae5d-ab87-4ba7-959e-aaf78f688e97-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:13.030106 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.030075    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerID="3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5" exitCode=0
Sep 11 03:56:13.030106 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.030100    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerID="933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7" exitCode=0
Sep 11 03:56:13.030288 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.030143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"}
Sep 11 03:56:13.030288 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.030222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"}
Sep 11 03:56:13.030288 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.030154    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.030288 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.030251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c23ae5d-ab87-4ba7-959e-aaf78f688e97","Type":"ContainerDied","Data":"4d387804c683d2310aa8ec49aa0177b04ebb1dbd98c7ffa17b36e1eadad4c0aa"}
Sep 11 03:56:13.030288 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.030284    2582 scope.go:117] "RemoveContainer" containerID="f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"
Sep 11 03:56:13.038642 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.038618    2582 scope.go:117] "RemoveContainer" containerID="3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"
Sep 11 03:56:13.046007 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.045985    2582 scope.go:117] "RemoveContainer" containerID="a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"
Sep 11 03:56:13.052928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.052910    2582 scope.go:117] "RemoveContainer" containerID="933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"
Sep 11 03:56:13.055063 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.055038    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:56:13.063626 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.060591    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:56:13.063626 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.061893    2582 scope.go:117] "RemoveContainer" containerID="3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"
Sep 11 03:56:13.070813 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.070793    2582 scope.go:117] "RemoveContainer" containerID="1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"
Sep 11 03:56:13.077840 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.077823    2582 scope.go:117] "RemoveContainer" containerID="333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3"
Sep 11 03:56:13.085493 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.085472    2582 scope.go:117] "RemoveContainer" containerID="f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"
Sep 11 03:56:13.085793 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:13.085769    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a\": container with ID starting with f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a not found: ID does not exist" containerID="f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"
Sep 11 03:56:13.085853 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.085803    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"} err="failed to get container status \"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a\": rpc error: code = NotFound desc = could not find container \"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a\": container with ID starting with f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a not found: ID does not exist"
Sep 11 03:56:13.085853 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.085824    2582 scope.go:117] "RemoveContainer" containerID="3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"
Sep 11 03:56:13.086107 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:13.086085    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5\": container with ID starting with 3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5 not found: ID does not exist" containerID="3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"
Sep 11 03:56:13.086189 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086112    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"} err="failed to get container status \"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5\": rpc error: code = NotFound desc = could not find container \"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5\": container with ID starting with 3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5 not found: ID does not exist"
Sep 11 03:56:13.086189 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086129    2582 scope.go:117] "RemoveContainer" containerID="a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"
Sep 11 03:56:13.086283 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086266    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:56:13.086378 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:13.086360    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db\": container with ID starting with a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db not found: ID does not exist" containerID="a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"
Sep 11 03:56:13.086419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086381    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"} err="failed to get container status \"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db\": rpc error: code = NotFound desc = could not find container \"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db\": container with ID starting with a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db not found: ID does not exist"
Sep 11 03:56:13.086419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086395    2582 scope.go:117] "RemoveContainer" containerID="933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"
Sep 11 03:56:13.086631 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:13.086609    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7\": container with ID starting with 933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7 not found: ID does not exist" containerID="933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"
Sep 11 03:56:13.086631 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086633    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"} err="failed to get container status \"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7\": rpc error: code = NotFound desc = could not find container \"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7\": container with ID starting with 933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7 not found: ID does not exist"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086646    2582 scope.go:117] "RemoveContainer" containerID="3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086655    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a3c269-b646-464a-a4a4-2780be3c3c31" containerName="console"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086669    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a3c269-b646-464a-a4a4-2780be3c3c31" containerName="console"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086678    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="init-config-reloader"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086684    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="init-config-reloader"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-web"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086697    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-web"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086706    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-metric"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086711    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-metric"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086722    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="prom-label-proxy"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086731    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="prom-label-proxy"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086742    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="config-reloader"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086747    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="config-reloader"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086755    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940fdf27-b346-4804-a8c9-a362daed6fa4" containerName="console"
Sep 11 03:56:13.086767 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086763    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="940fdf27-b346-4804-a8c9-a362daed6fa4" containerName="console"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086778    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="alertmanager"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086787    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="alertmanager"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086803    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086812    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086894    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="alertmanager"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086904    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-metric"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086912    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy-web"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:13.086904    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8\": container with ID starting with 3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8 not found: ID does not exist" containerID="3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086938    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"} err="failed to get container status \"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8\": rpc error: code = NotFound desc = could not find container \"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8\": container with ID starting with 3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8 not found: ID does not exist"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086959    2582 scope.go:117] "RemoveContainer" containerID="1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.086918    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="prom-label-proxy"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087009    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="kube-rbac-proxy"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087016    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" containerName="config-reloader"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087022    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="940fdf27-b346-4804-a8c9-a362daed6fa4" containerName="console"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087027    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a3c269-b646-464a-a4a4-2780be3c3c31" containerName="console"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:13.087222    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc\": container with ID starting with 1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc not found: ID does not exist" containerID="1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087245    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"} err="failed to get container status \"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc\": rpc error: code = NotFound desc = could not find container \"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc\": container with ID starting with 1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc not found: ID does not exist"
Sep 11 03:56:13.087350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087265    2582 scope.go:117] "RemoveContainer" containerID="333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3"
Sep 11 03:56:13.088013 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:13.087507    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3\": container with ID starting with 333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3 not found: ID does not exist" containerID="333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3"
Sep 11 03:56:13.088013 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087530    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3"} err="failed to get container status \"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3\": rpc error: code = NotFound desc = could not find container \"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3\": container with ID starting with 333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3 not found: ID does not exist"
Sep 11 03:56:13.088013 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087549    2582 scope.go:117] "RemoveContainer" containerID="f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"
Sep 11 03:56:13.088013 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087788    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a"} err="failed to get container status \"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a\": rpc error: code = NotFound desc = could not find container \"f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a\": container with ID starting with f018ef7df3ca05ab3b5f3a527ef494e998fb104268afa50e9705b01b41ee816a not found: ID does not exist"
Sep 11 03:56:13.088013 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.087809    2582 scope.go:117] "RemoveContainer" containerID="3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"
Sep 11 03:56:13.088188 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088035    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5"} err="failed to get container status \"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5\": rpc error: code = NotFound desc = could not find container \"3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5\": container with ID starting with 3bca5ccd96f3527d70fa050726262d981c5063f223c2e08b639e35d6a2a319e5 not found: ID does not exist"
Sep 11 03:56:13.088188 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088052    2582 scope.go:117] "RemoveContainer" containerID="a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"
Sep 11 03:56:13.088275 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088254    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db"} err="failed to get container status \"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db\": rpc error: code = NotFound desc = could not find container \"a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db\": container with ID starting with a6290bccf1e4871ad1863750331b454e616fa04a6ce3c73ec807316cbdf683db not found: ID does not exist"
Sep 11 03:56:13.088313 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088277    2582 scope.go:117] "RemoveContainer" containerID="933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"
Sep 11 03:56:13.088508 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088484    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7"} err="failed to get container status \"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7\": rpc error: code = NotFound desc = could not find container \"933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7\": container with ID starting with 933d59bee7ed75351527d9963a83a66e20673f299d445ec0f22537e6ab775ae7 not found: ID does not exist"
Sep 11 03:56:13.088544 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088509    2582 scope.go:117] "RemoveContainer" containerID="3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"
Sep 11 03:56:13.088770 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088751    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8"} err="failed to get container status \"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8\": rpc error: code = NotFound desc = could not find container \"3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8\": container with ID starting with 3f38fe14426139c9c949154efcc3c9acb70c093636dd513c9fb60a5d4083f6a8 not found: ID does not exist"
Sep 11 03:56:13.088819 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088771    2582 scope.go:117] "RemoveContainer" containerID="1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"
Sep 11 03:56:13.088986 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088967    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc"} err="failed to get container status \"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc\": rpc error: code = NotFound desc = could not find container \"1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc\": container with ID starting with 1607739f24c692cb38c719848cd4b916957ccf20f7151fa166263bf2d63b23bc not found: ID does not exist"
Sep 11 03:56:13.088986 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.088984    2582 scope.go:117] "RemoveContainer" containerID="333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3"
Sep 11 03:56:13.089226 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.089209    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3"} err="failed to get container status \"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3\": rpc error: code = NotFound desc = could not find container \"333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3\": container with ID starting with 333909af91e0c0d72e7ee888b317723540916c528eb836eb0a2a7b743e0d80b3 not found: ID does not exist"
Sep 11 03:56:13.091077 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.091056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.093793 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.093773    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 03:56:13.093892 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.093803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 03:56:13.093892 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.093803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6wmbg\""
Sep 11 03:56:13.094061 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.093909    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 03:56:13.094061 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.093953    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 03:56:13.094187 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.094173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 03:56:13.094312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.094292    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 03:56:13.094534 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.094518    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 03:56:13.095133 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.095116    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 03:56:13.099162 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.099143    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 03:56:13.103606 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.103586    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:56:13.184204 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184204 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6130d136-12f4-46f8-86f4-56086bcbebbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4qzh\" (UniqueName: \"kubernetes.io/projected/6130d136-12f4-46f8-86f4-56086bcbebbf-kube-api-access-p4qzh\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6130d136-12f4-46f8-86f4-56086bcbebbf-config-out\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6130d136-12f4-46f8-86f4-56086bcbebbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184413 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6130d136-12f4-46f8-86f4-56086bcbebbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184717 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184717 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6130d136-12f4-46f8-86f4-56086bcbebbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184717 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184717 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-web-config\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.184717 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.184639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285671 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-web-config\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285671 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285671 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285945 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6130d136-12f4-46f8-86f4-56086bcbebbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285945 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4qzh\" (UniqueName: \"kubernetes.io/projected/6130d136-12f4-46f8-86f4-56086bcbebbf-kube-api-access-p4qzh\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285945 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6130d136-12f4-46f8-86f4-56086bcbebbf-config-out\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285945 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.285945 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6130d136-12f4-46f8-86f4-56086bcbebbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.286223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.285953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.286223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.286008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6130d136-12f4-46f8-86f4-56086bcbebbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.286223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.286032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.286223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.286058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6130d136-12f4-46f8-86f4-56086bcbebbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.286223 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.286089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.286465 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.286286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6130d136-12f4-46f8-86f4-56086bcbebbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.286465 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.286452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6130d136-12f4-46f8-86f4-56086bcbebbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.288774 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.288607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6130d136-12f4-46f8-86f4-56086bcbebbf-config-out\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.288904 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.288868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.288995 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.288970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.289068 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.289055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.289129 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.289115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.289271 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.289249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.289390 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.289246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.289455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.289396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6130d136-12f4-46f8-86f4-56086bcbebbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.289515 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.289458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6130d136-12f4-46f8-86f4-56086bcbebbf-web-config\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.290772 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.290751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6130d136-12f4-46f8-86f4-56086bcbebbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.295589 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.295540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4qzh\" (UniqueName: \"kubernetes.io/projected/6130d136-12f4-46f8-86f4-56086bcbebbf-kube-api-access-p4qzh\") pod \"alertmanager-main-0\" (UID: \"6130d136-12f4-46f8-86f4-56086bcbebbf\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.400472 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.400433    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 03:56:13.680075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:13.680046    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 03:56:13.682321 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:56:13.682296    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6130d136_12f4_46f8_86f4_56086bcbebbf.slice/crio-d92dff7587c6a893db9739458ce8acdf7dfc41f11937c5c9abbfd6e6d2d13d29 WatchSource:0}: Error finding container d92dff7587c6a893db9739458ce8acdf7dfc41f11937c5c9abbfd6e6d2d13d29: Status 404 returned error can't find the container with id d92dff7587c6a893db9739458ce8acdf7dfc41f11937c5c9abbfd6e6d2d13d29
Sep 11 03:56:14.036055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:14.035957    2582 generic.go:356] "Generic (PLEG): container finished" podID="6130d136-12f4-46f8-86f4-56086bcbebbf" containerID="6e2f77218432bd4c0af58a9a8d0bd5fa69ae4364d2cfba902bc082f1ff11f11d" exitCode=0
Sep 11 03:56:14.036055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:14.036002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerDied","Data":"6e2f77218432bd4c0af58a9a8d0bd5fa69ae4364d2cfba902bc082f1ff11f11d"}
Sep 11 03:56:14.036257 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:14.036068    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerStarted","Data":"d92dff7587c6a893db9739458ce8acdf7dfc41f11937c5c9abbfd6e6d2d13d29"}
Sep 11 03:56:14.274770 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:14.274745    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c23ae5d-ab87-4ba7-959e-aaf78f688e97" path="/var/lib/kubelet/pods/4c23ae5d-ab87-4ba7-959e-aaf78f688e97/volumes"
Sep 11 03:56:15.042752 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.042717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerStarted","Data":"e6864dc415342874a11a4fb72037cb7961e1fb604ad2f453f5780920d58b765f"}
Sep 11 03:56:15.043102 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.042782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerStarted","Data":"49b8849864684d74440efe617c8e9435752e34be525773e72eb513691741a634"}
Sep 11 03:56:15.043102 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.042808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerStarted","Data":"ff5f50bdc6c7765b059c8e9af3b050da73bac5ce6ef53e01d30a1003051daaa5"}
Sep 11 03:56:15.043102 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.042836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerStarted","Data":"933f92760418383fbee0047a4638b8d92be9cfb310419ae289d1de4de2e0707f"}
Sep 11 03:56:15.043102 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.042865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerStarted","Data":"1b39bb62c4850b31a4f878369266ff6a86060480f27552cee4636f166a4f58d1"}
Sep 11 03:56:15.043102 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.042888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6130d136-12f4-46f8-86f4-56086bcbebbf","Type":"ContainerStarted","Data":"998e62ea67a52d5f918951bb483aaa3f2def667c795419ba929b661b41e2cb3d"}
Sep 11 03:56:15.070227 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.070026    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.054442985 podStartE2EDuration="2.070009967s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.566982ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:15.069851718 +0000 UTC m=+149.334330090" watchObservedRunningTime="2026-09-11 03:56:15.070009967 +0000 UTC m=+149.334488338"
Sep 11 03:56:15.276250 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.276216    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"]
Sep 11 03:56:15.278811 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.278785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.281448 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.281386    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 03:56:15.281448 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.281411    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 03:56:15.281651 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.281478    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 03:56:15.281651 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.281482    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 03:56:15.281758 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.281738    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xbm9r\""
Sep 11 03:56:15.281814 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.281769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 03:56:15.287743 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.287720    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 03:56:15.295046 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.294991    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"]
Sep 11 03:56:15.305748 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.305725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.305860 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.305772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-serving-certs-ca-bundle\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.305933 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.305888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-telemeter-client-tls\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.305933 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.305916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-federate-client-tls\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.306024 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.305939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59b7\" (UniqueName: \"kubernetes.io/projected/bd48999d-22f0-487b-ab80-17f843633e3f-kube-api-access-w59b7\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.306024 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.305965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-metrics-client-ca\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.306131 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.306099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-secret-telemeter-client\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.306182 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.306142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406615 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-serving-certs-ca-bundle\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406811 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-telemeter-client-tls\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406811 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-federate-client-tls\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406811 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59b7\" (UniqueName: \"kubernetes.io/projected/bd48999d-22f0-487b-ab80-17f843633e3f-kube-api-access-w59b7\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406976 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-metrics-client-ca\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406976 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-secret-telemeter-client\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406976 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.406976 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.406923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.407473 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.407445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-serving-certs-ca-bundle\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.407718 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.407695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-metrics-client-ca\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.408193 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.408169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd48999d-22f0-487b-ab80-17f843633e3f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.409572 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.409502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-telemeter-client-tls\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.409690 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.409667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.409735 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.409669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-federate-client-tls\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.409785 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.409767    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd48999d-22f0-487b-ab80-17f843633e3f-secret-telemeter-client\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.414935 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.414914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59b7\" (UniqueName: \"kubernetes.io/projected/bd48999d-22f0-487b-ab80-17f843633e3f-kube-api-access-w59b7\") pod \"telemeter-client-d97fb68cf-9qq2m\" (UID: \"bd48999d-22f0-487b-ab80-17f843633e3f\") " pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.569431 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.569339    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:56:15.569762 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.569704    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="prometheus" containerID="cri-o://2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e" gracePeriod=600
Sep 11 03:56:15.569762 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.569726    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy" containerID="cri-o://a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436" gracePeriod=600
Sep 11 03:56:15.569943 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.569763    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-web" containerID="cri-o://5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e" gracePeriod=600
Sep 11 03:56:15.569943 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.569777    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="config-reloader" containerID="cri-o://399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226" gracePeriod=600
Sep 11 03:56:15.569943 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.569741    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="thanos-sidecar" containerID="cri-o://c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a" gracePeriod=600
Sep 11 03:56:15.569943 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.569787    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1" gracePeriod=600
Sep 11 03:56:15.589887 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.589864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"
Sep 11 03:56:15.882624 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:15.882527    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d97fb68cf-9qq2m"]
Sep 11 03:56:15.886275 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:56:15.886247    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd48999d_22f0_487b_ab80_17f843633e3f.slice/crio-5c442b1a06c0b9c00b11e23508dfa5dfdeb6f2cc7797b2ebb3b932f298fb1f9c WatchSource:0}: Error finding container 5c442b1a06c0b9c00b11e23508dfa5dfdeb6f2cc7797b2ebb3b932f298fb1f9c: Status 404 returned error can't find the container with id 5c442b1a06c0b9c00b11e23508dfa5dfdeb6f2cc7797b2ebb3b932f298fb1f9c
Sep 11 03:56:16.050953 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.050917    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerID="d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1" exitCode=0
Sep 11 03:56:16.050953 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.050941    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerID="a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436" exitCode=0
Sep 11 03:56:16.050953 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.050947    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerID="c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a" exitCode=0
Sep 11 03:56:16.050953 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.050953    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerID="399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226" exitCode=0
Sep 11 03:56:16.050953 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.050958    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerID="2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e" exitCode=0
Sep 11 03:56:16.051461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.050986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1"}
Sep 11 03:56:16.051461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.051058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436"}
Sep 11 03:56:16.051461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.051087    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a"}
Sep 11 03:56:16.051461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.051115    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226"}
Sep 11 03:56:16.051461 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.051147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e"}
Sep 11 03:56:16.052249 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.052226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m" event={"ID":"bd48999d-22f0-487b-ab80-17f843633e3f","Type":"ContainerStarted","Data":"5c442b1a06c0b9c00b11e23508dfa5dfdeb6f2cc7797b2ebb3b932f298fb1f9c"}
Sep 11 03:56:16.816419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.816396    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:16.920818 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.920789    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-web-config\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.920977 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.920833    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-kubelet-serving-ca-bundle\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.920977 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.920870    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-trusted-ca-bundle\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.920977 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.920893    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvnts\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-kube-api-access-rvnts\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.920977 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.920927    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-thanos-prometheus-http-client-file\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.920977 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.920951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-serving-certs-ca-bundle\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.920977 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.920972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-metrics-client-ca\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921032    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-1\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921058    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921086    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-metrics-client-certs\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-0\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-tls\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config-out\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921202    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-tls-assets\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921649 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-kube-rbac-proxy\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921745    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-db\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921803    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921865    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-grpc-tls\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.921928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.921918    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-2\") pod \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\" (UID: \"0cf59fa8-d04b-45c3-8fce-364fa08bcad3\") "
Sep 11 03:56:16.922735 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.922704    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:16.924115 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.923510    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:16.924910 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.924879    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:16.926494 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.925726    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:16.926941 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.926693    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:16.927587 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.927539    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:16.931418 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.928870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:16.931418 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.929436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:16.931418 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.929580    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:16.932605 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.932245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config-out" (OuterVolumeSpecName: "config-out") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:56:16.937683 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.937628    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.938325 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.938283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-kube-api-access-rvnts" (OuterVolumeSpecName: "kube-api-access-rvnts") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "kube-api-access-rvnts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:16.938505 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.938435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.938608 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.938584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config" (OuterVolumeSpecName: "config") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.938782 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.938729    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.938898 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.938868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.939023 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.939003    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.939104 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.939084    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.939254 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.939231    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:16.947858 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:16.947832    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-web-config" (OuterVolumeSpecName: "web-config") pod "0cf59fa8-d04b-45c3-8fce-364fa08bcad3" (UID: "0cf59fa8-d04b-45c3-8fce-364fa08bcad3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:17.024003 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.023973    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-web-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024003 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024004    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024020    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024039    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvnts\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-kube-api-access-rvnts\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024056    2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024068    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024081    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024095    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-configmap-metrics-client-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024107    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024119    2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024131    2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-metrics-client-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024143    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024155    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024168    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-config-out\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024182    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-tls-assets\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024194    2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-kube-rbac-proxy\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024207    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-db\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024216 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024220    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024709 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024234    2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-secret-grpc-tls\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.024709 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.024247    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cf59fa8-d04b-45c3-8fce-364fa08bcad3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:17.060334 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.060302    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerID="5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e" exitCode=0
Sep 11 03:56:17.060800 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.060380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e"}
Sep 11 03:56:17.060800 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.060399    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.060800 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.060476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0cf59fa8-d04b-45c3-8fce-364fa08bcad3","Type":"ContainerDied","Data":"92411f2aed6e5b0aeb7fb4c277400e5bee6f0390bd7c0afbaa071f3af409b72f"}
Sep 11 03:56:17.060800 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.060509    2582 scope.go:117] "RemoveContainer" containerID="d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1"
Sep 11 03:56:17.084676 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.084650    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:56:17.089371 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.089351    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:56:17.120656 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.120621    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:56:17.120991 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.120975    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-thanos"
Sep 11 03:56:17.120991 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.120992    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-thanos"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121003    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-web"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121009    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-web"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121018    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="prometheus"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121024    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="prometheus"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121039    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="thanos-sidecar"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121047    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="thanos-sidecar"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121063    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="config-reloader"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121072    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="config-reloader"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121082    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="init-config-reloader"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121087    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="init-config-reloader"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121102    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy"
Sep 11 03:56:17.121112 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121107    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy"
Sep 11 03:56:17.121670 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121158    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="config-reloader"
Sep 11 03:56:17.121670 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121177    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="thanos-sidecar"
Sep 11 03:56:17.121670 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121185    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="prometheus"
Sep 11 03:56:17.121670 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121191    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy"
Sep 11 03:56:17.121670 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121196    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-thanos"
Sep 11 03:56:17.121670 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.121201    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" containerName="kube-rbac-proxy-web"
Sep 11 03:56:17.124691 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.124670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.127520 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.127500    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 03:56:17.127646 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.127524    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 03:56:17.127868 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.127849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 03:56:17.127949 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.127880    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 03:56:17.127949 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.127941    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 03:56:17.128055 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.127997    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sq7dp\""
Sep 11 03:56:17.128115 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128066    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ajhbijbpkpmo\""
Sep 11 03:56:17.128115 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128083    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 03:56:17.128374 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128355    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 03:56:17.128442 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128399    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 03:56:17.128527 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128361    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 03:56:17.128623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128607    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 03:56:17.128699 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128686    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 03:56:17.128761 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.128737    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 03:56:17.136810 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.136599    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 03:56:17.140475 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.140451    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 03:56:17.180430 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.180405    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:56:17.226265 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226265 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkf59\" (UniqueName: \"kubernetes.io/projected/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-kube-api-access-pkf59\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226455 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-config\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.226763 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.227092 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.227092 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.226896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328293 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328293 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328525 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328525 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328525 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328525 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328525 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328525 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkf59\" (UniqueName: \"kubernetes.io/projected/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-kube-api-access-pkf59\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-config\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.328856 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.329312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.329312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.328922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.329312 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.329295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.330736 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.329481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.330736 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.330390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.331575 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.331542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.332058 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.332014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.332181 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.332083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.332181 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.332102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333286 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.332648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333286 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.332737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333286 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.333011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333286 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.333181    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333540 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.333388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333723 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.333660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333723 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.333673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.333954 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.333931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.334622 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.334598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-config\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.334979 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.334960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.335440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.335422    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.336426 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.336405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.337505 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.337483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkf59\" (UniqueName: \"kubernetes.io/projected/e4455fa2-c2e2-4127-8da7-0ecb2abed9dd-kube-api-access-pkf59\") pod \"prometheus-k8s-0\" (UID: \"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.436740 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.436710    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:17.504678 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.504651    2582 scope.go:117] "RemoveContainer" containerID="a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436"
Sep 11 03:56:17.527491 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.527471    2582 scope.go:117] "RemoveContainer" containerID="5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e"
Sep 11 03:56:17.535213 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.535191    2582 scope.go:117] "RemoveContainer" containerID="c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a"
Sep 11 03:56:17.543237 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.543204    2582 scope.go:117] "RemoveContainer" containerID="399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226"
Sep 11 03:56:17.555199 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.555154    2582 scope.go:117] "RemoveContainer" containerID="2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e"
Sep 11 03:56:17.568257 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.568241    2582 scope.go:117] "RemoveContainer" containerID="fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a"
Sep 11 03:56:17.587098 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.587073    2582 scope.go:117] "RemoveContainer" containerID="d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1"
Sep 11 03:56:17.587392 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:17.587373    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1\": container with ID starting with d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1 not found: ID does not exist" containerID="d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1"
Sep 11 03:56:17.587435 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.587402    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1"} err="failed to get container status \"d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1\": rpc error: code = NotFound desc = could not find container \"d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1\": container with ID starting with d5d265caaf57d6936cedde74b3273fdd2a87ddfb6e510dd20a6455bdfad25dd1 not found: ID does not exist"
Sep 11 03:56:17.587435 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.587420    2582 scope.go:117] "RemoveContainer" containerID="a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436"
Sep 11 03:56:17.587718 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:17.587694    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436\": container with ID starting with a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436 not found: ID does not exist" containerID="a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436"
Sep 11 03:56:17.587772 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.587726    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436"} err="failed to get container status \"a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436\": rpc error: code = NotFound desc = could not find container \"a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436\": container with ID starting with a0000147b4a7e526ed08f8091a1f567e65b20d62aec2ae0c97c2f4ac12e28436 not found: ID does not exist"
Sep 11 03:56:17.587772 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.587749    2582 scope.go:117] "RemoveContainer" containerID="5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e"
Sep 11 03:56:17.588089 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:17.588065    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e\": container with ID starting with 5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e not found: ID does not exist" containerID="5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e"
Sep 11 03:56:17.588150 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.588096    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e"} err="failed to get container status \"5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e\": rpc error: code = NotFound desc = could not find container \"5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e\": container with ID starting with 5604990102b8a9939303c814225a75c9efa2c1a42b47199de9548654abe8ae5e not found: ID does not exist"
Sep 11 03:56:17.588150 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.588116    2582 scope.go:117] "RemoveContainer" containerID="c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a"
Sep 11 03:56:17.588441 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:17.588423    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a\": container with ID starting with c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a not found: ID does not exist" containerID="c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a"
Sep 11 03:56:17.588493 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.588447    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a"} err="failed to get container status \"c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a\": rpc error: code = NotFound desc = could not find container \"c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a\": container with ID starting with c7a2eb5d14c587b6da6898c3569538637085471f50a69b4529fb31ee7044130a not found: ID does not exist"
Sep 11 03:56:17.588493 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.588463    2582 scope.go:117] "RemoveContainer" containerID="399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226"
Sep 11 03:56:17.588722 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:17.588703    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226\": container with ID starting with 399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226 not found: ID does not exist" containerID="399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226"
Sep 11 03:56:17.588793 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.588729    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226"} err="failed to get container status \"399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226\": rpc error: code = NotFound desc = could not find container \"399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226\": container with ID starting with 399d0d62f5ef59350d6de9c42da1d24a0b257aa6eabc925c4fd2c006e6272226 not found: ID does not exist"
Sep 11 03:56:17.588793 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.588748    2582 scope.go:117] "RemoveContainer" containerID="2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e"
Sep 11 03:56:17.589067 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:17.589043    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e\": container with ID starting with 2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e not found: ID does not exist" containerID="2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e"
Sep 11 03:56:17.589109 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.589071    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e"} err="failed to get container status \"2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e\": rpc error: code = NotFound desc = could not find container \"2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e\": container with ID starting with 2f3a7ab8eb5fc4fa9c62d7d05d23eef2cbcacfbd3cb47ccbb561980c80310d6e not found: ID does not exist"
Sep 11 03:56:17.589109 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.589085    2582 scope.go:117] "RemoveContainer" containerID="fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a"
Sep 11 03:56:17.589351 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:17.589334    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a\": container with ID starting with fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a not found: ID does not exist" containerID="fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a"
Sep 11 03:56:17.589422 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.589358    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a"} err="failed to get container status \"fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a\": rpc error: code = NotFound desc = could not find container \"fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a\": container with ID starting with fa2f94e007615918b34e8d969cc3c1a464a4f6ea887fc0231f0ed08f06fcf46a not found: ID does not exist"
Sep 11 03:56:17.783236 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:17.783213    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 03:56:17.785707 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:56:17.785667    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4455fa2_c2e2_4127_8da7_0ecb2abed9dd.slice/crio-64316a5f0b780a91ad5ef08788875d25b0b7f20a06feb672ea5e9e01ba42a16d WatchSource:0}: Error finding container 64316a5f0b780a91ad5ef08788875d25b0b7f20a06feb672ea5e9e01ba42a16d: Status 404 returned error can't find the container with id 64316a5f0b780a91ad5ef08788875d25b0b7f20a06feb672ea5e9e01ba42a16d
Sep 11 03:56:18.067133 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.067045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m" event={"ID":"bd48999d-22f0-487b-ab80-17f843633e3f","Type":"ContainerStarted","Data":"375a4267a03a78f1060dfec6f6bad4230adf98b97be368e113481236405c246e"}
Sep 11 03:56:18.067133 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.067115    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m" event={"ID":"bd48999d-22f0-487b-ab80-17f843633e3f","Type":"ContainerStarted","Data":"adc3f349900c9eba50a599ca625b03a616b8d7d64375da354e9255a82b82b59c"}
Sep 11 03:56:18.067599 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.067147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m" event={"ID":"bd48999d-22f0-487b-ab80-17f843633e3f","Type":"ContainerStarted","Data":"b0564490ffcd4d523318d8ff58f9415417f282fc0c70d2760c2756333223ec06"}
Sep 11 03:56:18.068549 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.068525    2582 generic.go:356] "Generic (PLEG): container finished" podID="e4455fa2-c2e2-4127-8da7-0ecb2abed9dd" containerID="3406e916f1a2d92a2f19c2321ebc0c7e6d18e15e876a2b015e98d058698e4534" exitCode=0
Sep 11 03:56:18.068656 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.068605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerDied","Data":"3406e916f1a2d92a2f19c2321ebc0c7e6d18e15e876a2b015e98d058698e4534"}
Sep 11 03:56:18.068710 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.068656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerStarted","Data":"64316a5f0b780a91ad5ef08788875d25b0b7f20a06feb672ea5e9e01ba42a16d"}
Sep 11 03:56:18.088419 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.088369    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d97fb68cf-9qq2m" podStartSLOduration=1.4404944450000001 podStartE2EDuration="3.088350027s" totalImagesPullingTime="1.647855582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:18.087090418 +0000 UTC m=+152.351568790" watchObservedRunningTime="2026-09-11 03:56:18.088350027 +0000 UTC m=+152.352828399"
Sep 11 03:56:18.174501 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.174467    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-686ff5d8f-fjm99"]
Sep 11 03:56:18.177103 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.177078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.191252 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.191226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686ff5d8f-fjm99"]
Sep 11 03:56:18.238266 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.238051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-oauth-config\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.238266 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.238092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbvm\" (UniqueName: \"kubernetes.io/projected/23cf6241-2180-4fba-98d0-812c7dcba384-kube-api-access-kdbvm\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.238266 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.238115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-console-config\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.238266 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.238134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-oauth-serving-cert\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.238266 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.238186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-trusted-ca-bundle\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.238266 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.238246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-serving-cert\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.238550 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.238271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-service-ca\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.279883 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.279849    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf59fa8-d04b-45c3-8fce-364fa08bcad3" path="/var/lib/kubelet/pods/0cf59fa8-d04b-45c3-8fce-364fa08bcad3/volumes"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.339443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-trusted-ca-bundle\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.340130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-serving-cert\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.340171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-service-ca\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.340233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-oauth-config\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.340286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbvm\" (UniqueName: \"kubernetes.io/projected/23cf6241-2180-4fba-98d0-812c7dcba384-kube-api-access-kdbvm\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.340327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-console-config\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.340327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-trusted-ca-bundle\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.340576 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.340350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-oauth-serving-cert\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.341071 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.341049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-oauth-serving-cert\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.341420 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.341371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-console-config\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.341712 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.341694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-service-ca\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.342899 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.342876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-serving-cert\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.343197 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.343175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-oauth-config\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.351249 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.351226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbvm\" (UniqueName: \"kubernetes.io/projected/23cf6241-2180-4fba-98d0-812c7dcba384-kube-api-access-kdbvm\") pod \"console-686ff5d8f-fjm99\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") " pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.490900 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.490694    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:18.772078 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:18.772046    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686ff5d8f-fjm99"]
Sep 11 03:56:18.774645 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:56:18.774610    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cf6241_2180_4fba_98d0_812c7dcba384.slice/crio-8deee8ee728237475807815fabc9e50d358d73aa3424512bf3e49564e50bf469 WatchSource:0}: Error finding container 8deee8ee728237475807815fabc9e50d358d73aa3424512bf3e49564e50bf469: Status 404 returned error can't find the container with id 8deee8ee728237475807815fabc9e50d358d73aa3424512bf3e49564e50bf469
Sep 11 03:56:19.076273 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.076172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686ff5d8f-fjm99" event={"ID":"23cf6241-2180-4fba-98d0-812c7dcba384","Type":"ContainerStarted","Data":"098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7"}
Sep 11 03:56:19.076273 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.076228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686ff5d8f-fjm99" event={"ID":"23cf6241-2180-4fba-98d0-812c7dcba384","Type":"ContainerStarted","Data":"8deee8ee728237475807815fabc9e50d358d73aa3424512bf3e49564e50bf469"}
Sep 11 03:56:19.079546 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.079516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerStarted","Data":"f573c71126a7438959089913a4f5e7bc176d17e0c8295081a25900275affc603"}
Sep 11 03:56:19.079697 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.079607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerStarted","Data":"18f03f415d492e5c047bd5ce8dcb8913756c010fc640e81bcd83e4e7faaaaa0e"}
Sep 11 03:56:19.079697 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.079656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerStarted","Data":"5ed61823255de70ef30d43ee9e5df8bc34a89722020a684011a11e1eb023721a"}
Sep 11 03:56:19.079697 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.079692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerStarted","Data":"52fa43d12f2733c05ff608e725bc2346f11a82b4d8cf5aacae285e9d8a346e9d"}
Sep 11 03:56:19.079850 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.079731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerStarted","Data":"0c4b17ae72b9d26274dd9599e456a9406e7f4a2f210745f2170cdadd5085169e"}
Sep 11 03:56:19.079850 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.079778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4455fa2-c2e2-4127-8da7-0ecb2abed9dd","Type":"ContainerStarted","Data":"07bb27c7d8fe06853313cf532b3d8a3a36800194ac9063677f25e14996b38d1d"}
Sep 11 03:56:19.093329 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.093286    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-686ff5d8f-fjm99" podStartSLOduration=1.093271214 podStartE2EDuration="1.093271214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:19.091952948 +0000 UTC m=+153.356431319" watchObservedRunningTime="2026-09-11 03:56:19.093271214 +0000 UTC m=+153.357749585"
Sep 11 03:56:19.121262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:19.121213    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.101154567 podStartE2EDuration="2.121193988s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.039421ms" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:56:19.117367516 +0000 UTC m=+153.381845887" watchObservedRunningTime="2026-09-11 03:56:19.121193988 +0000 UTC m=+153.385672360"
Sep 11 03:56:22.437284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:22.437255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:56:28.491679 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:28.491640    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:28.492110 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:28.491703    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:28.496490 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:28.496469    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:29.126067 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:29.126038    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:56:29.165828 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:29.165796    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f87949f6-pnc8k"]
Sep 11 03:56:54.186486 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.186422    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66f87949f6-pnc8k" podUID="46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" containerName="console" containerID="cri-o://591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739" gracePeriod=15
Sep 11 03:56:54.422943 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.422917    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:56:54.557773 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.557691    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbnpw\" (UniqueName: \"kubernetes.io/projected/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-kube-api-access-vbnpw\") pod \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") "
Sep 11 03:56:54.557773 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.557760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-oauth-config\") pod \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") "
Sep 11 03:56:54.557983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.557790    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-oauth-serving-cert\") pod \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") "
Sep 11 03:56:54.557983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.557828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-trusted-ca-bundle\") pod \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") "
Sep 11 03:56:54.557983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.557864    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-serving-cert\") pod \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") "
Sep 11 03:56:54.557983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.557884    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-service-ca\") pod \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") "
Sep 11 03:56:54.557983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.557916    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-config\") pod \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\" (UID: \"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b\") "
Sep 11 03:56:54.558334 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558280    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" (UID: "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:54.558334 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-service-ca" (OuterVolumeSpecName: "service-ca") pod "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" (UID: "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:54.558536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558319    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" (UID: "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:54.558536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558326    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-config" (OuterVolumeSpecName: "console-config") pod "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" (UID: "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:56:54.558536 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558488    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-oauth-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:54.558731 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558541    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-trusted-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:54.558731 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558575    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:54.558731 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.558588    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:54.560021 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.559995    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-kube-api-access-vbnpw" (OuterVolumeSpecName: "kube-api-access-vbnpw") pod "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" (UID: "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b"). InnerVolumeSpecName "kube-api-access-vbnpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:56:54.560119 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.560104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" (UID: "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:54.560173 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.560157    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" (UID: "46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:56:54.659845 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.659815    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-oauth-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:54.659845 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.659841    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-console-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:54.660023 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:54.659851    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbnpw\" (UniqueName: \"kubernetes.io/projected/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b-kube-api-access-vbnpw\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:56:55.230836 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.230800    2582 generic.go:356] "Generic (PLEG): container finished" podID="46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" containerID="591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739" exitCode=0
Sep 11 03:56:55.231231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.230847    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f87949f6-pnc8k"
Sep 11 03:56:55.231231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.230883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f87949f6-pnc8k" event={"ID":"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b","Type":"ContainerDied","Data":"591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739"}
Sep 11 03:56:55.231231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.230963    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f87949f6-pnc8k" event={"ID":"46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b","Type":"ContainerDied","Data":"057ae9be66d0749347cf1cca8f7595f3aeacb15c5aca8d13aeb8b924c31e6bd8"}
Sep 11 03:56:55.231231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.230983    2582 scope.go:117] "RemoveContainer" containerID="591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739"
Sep 11 03:56:55.239547 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.239532    2582 scope.go:117] "RemoveContainer" containerID="591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739"
Sep 11 03:56:55.239821 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:56:55.239803    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739\": container with ID starting with 591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739 not found: ID does not exist" containerID="591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739"
Sep 11 03:56:55.239870 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.239830    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739"} err="failed to get container status \"591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739\": rpc error: code = NotFound desc = could not find container \"591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739\": container with ID starting with 591ed510052b45f3e0c8cb4df87ca494faa79c6ec806e9bb1a78df8c4afeb739 not found: ID does not exist"
Sep 11 03:56:55.252424 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.252397    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f87949f6-pnc8k"]
Sep 11 03:56:55.255983 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:55.255963    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66f87949f6-pnc8k"]
Sep 11 03:56:56.273802 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:56:56.273768    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" path="/var/lib/kubelet/pods/46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b/volumes"
Sep 11 03:57:17.437350 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:17.437311    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:57:17.452738 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:17.452704    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:57:18.341215 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:18.341187    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 03:57:28.663231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.663191    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-589764645d-cc26k"]
Sep 11 03:57:28.663747 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.663731    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" containerName="console"
Sep 11 03:57:28.663812 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.663751    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" containerName="console"
Sep 11 03:57:28.663866 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.663826    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ea8b90-1bf1-4a3a-bc8c-ae6cd506ad8b" containerName="console"
Sep 11 03:57:28.668206 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.668185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.685149 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.685117    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-589764645d-cc26k"]
Sep 11 03:57:28.745804 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.745763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-oauth-config\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.746009 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.745830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-service-ca\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.746009 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.745906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8cw\" (UniqueName: \"kubernetes.io/projected/8d017451-ced0-4fe7-9332-0d6a725af531-kube-api-access-jk8cw\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.746009 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.745958    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-console-config\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.746009 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.745998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-serving-cert\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.746194 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.746045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-oauth-serving-cert\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.746194 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.746078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-trusted-ca-bundle\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.846851 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.846797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-oauth-serving-cert\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.846851 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.846851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-trusted-ca-bundle\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847101 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.846880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-oauth-config\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847101 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.846918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-service-ca\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847101 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.846936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8cw\" (UniqueName: \"kubernetes.io/projected/8d017451-ced0-4fe7-9332-0d6a725af531-kube-api-access-jk8cw\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847101 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.846972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-console-config\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847101 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.847001    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-serving-cert\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847712 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.847686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-oauth-serving-cert\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847712 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.847704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-console-config\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847883 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.847736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-service-ca\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.847883 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.847808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-trusted-ca-bundle\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.849398 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.849378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-serving-cert\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.849486 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.849400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-oauth-config\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.855099 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.855072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8cw\" (UniqueName: \"kubernetes.io/projected/8d017451-ced0-4fe7-9332-0d6a725af531-kube-api-access-jk8cw\") pod \"console-589764645d-cc26k\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") " pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:28.977353 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:28.977302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:29.250861 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:29.250767    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-589764645d-cc26k"]
Sep 11 03:57:29.253940 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:57:29.253912    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d017451_ced0_4fe7_9332_0d6a725af531.slice/crio-6a980671fd243f17d2c734a8ed86d8ba961ab545d7c3f088f7d1ca58743fb025 WatchSource:0}: Error finding container 6a980671fd243f17d2c734a8ed86d8ba961ab545d7c3f088f7d1ca58743fb025: Status 404 returned error can't find the container with id 6a980671fd243f17d2c734a8ed86d8ba961ab545d7c3f088f7d1ca58743fb025
Sep 11 03:57:29.372918 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:29.372882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589764645d-cc26k" event={"ID":"8d017451-ced0-4fe7-9332-0d6a725af531","Type":"ContainerStarted","Data":"1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e"}
Sep 11 03:57:29.373075 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:29.372936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589764645d-cc26k" event={"ID":"8d017451-ced0-4fe7-9332-0d6a725af531","Type":"ContainerStarted","Data":"6a980671fd243f17d2c734a8ed86d8ba961ab545d7c3f088f7d1ca58743fb025"}
Sep 11 03:57:29.395325 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:29.394968    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-589764645d-cc26k" podStartSLOduration=1.394944722 podStartE2EDuration="1.394944722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:29.394330128 +0000 UTC m=+223.658808525" watchObservedRunningTime="2026-09-11 03:57:29.394944722 +0000 UTC m=+223.659423094"
Sep 11 03:57:38.978332 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:38.978293    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:38.978862 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:38.978374    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:38.982735 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:38.982717    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:39.420477 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:39.420446    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-589764645d-cc26k"
Sep 11 03:57:39.466273 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:57:39.466224    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-686ff5d8f-fjm99"]
Sep 11 03:58:04.485663 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.485626    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-686ff5d8f-fjm99" podUID="23cf6241-2180-4fba-98d0-812c7dcba384" containerName="console" containerID="cri-o://098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7" gracePeriod=15
Sep 11 03:58:04.727705 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.727682    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:58:04.763353 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763282    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-serving-cert\") pod \"23cf6241-2180-4fba-98d0-812c7dcba384\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") "
Sep 11 03:58:04.763353 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763317    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-service-ca\") pod \"23cf6241-2180-4fba-98d0-812c7dcba384\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") "
Sep 11 03:58:04.763353 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763346    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-oauth-serving-cert\") pod \"23cf6241-2180-4fba-98d0-812c7dcba384\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") "
Sep 11 03:58:04.763648 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763366    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-trusted-ca-bundle\") pod \"23cf6241-2180-4fba-98d0-812c7dcba384\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") "
Sep 11 03:58:04.763648 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763495    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdbvm\" (UniqueName: \"kubernetes.io/projected/23cf6241-2180-4fba-98d0-812c7dcba384-kube-api-access-kdbvm\") pod \"23cf6241-2180-4fba-98d0-812c7dcba384\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") "
Sep 11 03:58:04.763648 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-console-config\") pod \"23cf6241-2180-4fba-98d0-812c7dcba384\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") "
Sep 11 03:58:04.763648 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763628    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-oauth-config\") pod \"23cf6241-2180-4fba-98d0-812c7dcba384\" (UID: \"23cf6241-2180-4fba-98d0-812c7dcba384\") "
Sep 11 03:58:04.763866 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763739    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "23cf6241-2180-4fba-98d0-812c7dcba384" (UID: "23cf6241-2180-4fba-98d0-812c7dcba384"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:58:04.763866 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763755    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-service-ca" (OuterVolumeSpecName: "service-ca") pod "23cf6241-2180-4fba-98d0-812c7dcba384" (UID: "23cf6241-2180-4fba-98d0-812c7dcba384"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:58:04.763866 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.763762    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "23cf6241-2180-4fba-98d0-812c7dcba384" (UID: "23cf6241-2180-4fba-98d0-812c7dcba384"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:58:04.764023 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.764006    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-console-config" (OuterVolumeSpecName: "console-config") pod "23cf6241-2180-4fba-98d0-812c7dcba384" (UID: "23cf6241-2180-4fba-98d0-812c7dcba384"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:58:04.764194 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.764174    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:58:04.764277 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.764196    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-oauth-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:58:04.764277 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.764206    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-trusted-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:58:04.764277 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.764214    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23cf6241-2180-4fba-98d0-812c7dcba384-console-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:58:04.765919 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.765897    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "23cf6241-2180-4fba-98d0-812c7dcba384" (UID: "23cf6241-2180-4fba-98d0-812c7dcba384"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:58:04.766037 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.766020    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "23cf6241-2180-4fba-98d0-812c7dcba384" (UID: "23cf6241-2180-4fba-98d0-812c7dcba384"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:58:04.766119 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.766103    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cf6241-2180-4fba-98d0-812c7dcba384-kube-api-access-kdbvm" (OuterVolumeSpecName: "kube-api-access-kdbvm") pod "23cf6241-2180-4fba-98d0-812c7dcba384" (UID: "23cf6241-2180-4fba-98d0-812c7dcba384"). InnerVolumeSpecName "kube-api-access-kdbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:04.865088 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.865051    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:58:04.865088 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.865082    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbvm\" (UniqueName: \"kubernetes.io/projected/23cf6241-2180-4fba-98d0-812c7dcba384-kube-api-access-kdbvm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:58:04.865088 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:04.865092    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23cf6241-2180-4fba-98d0-812c7dcba384-console-oauth-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 03:58:05.521765 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.521724    2582 generic.go:356] "Generic (PLEG): container finished" podID="23cf6241-2180-4fba-98d0-812c7dcba384" containerID="098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7" exitCode=0
Sep 11 03:58:05.522226 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.521777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686ff5d8f-fjm99" event={"ID":"23cf6241-2180-4fba-98d0-812c7dcba384","Type":"ContainerDied","Data":"098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7"}
Sep 11 03:58:05.522226 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.521816    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686ff5d8f-fjm99" event={"ID":"23cf6241-2180-4fba-98d0-812c7dcba384","Type":"ContainerDied","Data":"8deee8ee728237475807815fabc9e50d358d73aa3424512bf3e49564e50bf469"}
Sep 11 03:58:05.522226 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.521836    2582 scope.go:117] "RemoveContainer" containerID="098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7"
Sep 11 03:58:05.522226 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.521778    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686ff5d8f-fjm99"
Sep 11 03:58:05.530220 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.530203    2582 scope.go:117] "RemoveContainer" containerID="098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7"
Sep 11 03:58:05.530503 ip-10-0-138-179 kubenswrapper[2582]: E0911 03:58:05.530483    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7\": container with ID starting with 098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7 not found: ID does not exist" containerID="098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7"
Sep 11 03:58:05.530579 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.530509    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7"} err="failed to get container status \"098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7\": rpc error: code = NotFound desc = could not find container \"098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7\": container with ID starting with 098baab2f86c579bd57482475b684cc936ee0793008e912e9ad1edaf333245b7 not found: ID does not exist"
Sep 11 03:58:05.541658 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.541632    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-686ff5d8f-fjm99"]
Sep 11 03:58:05.547246 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:05.547222    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-686ff5d8f-fjm99"]
Sep 11 03:58:06.273928 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:06.273894    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23cf6241-2180-4fba-98d0-812c7dcba384" path="/var/lib/kubelet/pods/23cf6241-2180-4fba-98d0-812c7dcba384/volumes"
Sep 11 03:58:46.167607 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:46.167576    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 03:58:46.171997 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:46.171970    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 03:58:46.173998 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:58:46.173978    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 03:59:20.364773 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.364735    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jkmw9"]
Sep 11 03:59:20.365231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.365070    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23cf6241-2180-4fba-98d0-812c7dcba384" containerName="console"
Sep 11 03:59:20.365231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.365081    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="23cf6241-2180-4fba-98d0-812c7dcba384" containerName="console"
Sep 11 03:59:20.365231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.365143    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="23cf6241-2180-4fba-98d0-812c7dcba384" containerName="console"
Sep 11 03:59:20.368043 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.368026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.370782 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.370758    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 03:59:20.370902 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.370830    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-szqc2\""
Sep 11 03:59:20.370902 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.370838    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 03:59:20.378321 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.378297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jkmw9"]
Sep 11 03:59:20.516434 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.516398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpc66\" (UniqueName: \"kubernetes.io/projected/9939ede7-5461-428d-86db-cd66a44cac59-kube-api-access-dpc66\") pod \"cert-manager-55b4d954d5-jkmw9\" (UID: \"9939ede7-5461-428d-86db-cd66a44cac59\") " pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.516633 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.516529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9939ede7-5461-428d-86db-cd66a44cac59-bound-sa-token\") pod \"cert-manager-55b4d954d5-jkmw9\" (UID: \"9939ede7-5461-428d-86db-cd66a44cac59\") " pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.618027 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.617932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9939ede7-5461-428d-86db-cd66a44cac59-bound-sa-token\") pod \"cert-manager-55b4d954d5-jkmw9\" (UID: \"9939ede7-5461-428d-86db-cd66a44cac59\") " pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.618027 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.617982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpc66\" (UniqueName: \"kubernetes.io/projected/9939ede7-5461-428d-86db-cd66a44cac59-kube-api-access-dpc66\") pod \"cert-manager-55b4d954d5-jkmw9\" (UID: \"9939ede7-5461-428d-86db-cd66a44cac59\") " pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.626621 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.626592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9939ede7-5461-428d-86db-cd66a44cac59-bound-sa-token\") pod \"cert-manager-55b4d954d5-jkmw9\" (UID: \"9939ede7-5461-428d-86db-cd66a44cac59\") " pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.626883 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.626860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpc66\" (UniqueName: \"kubernetes.io/projected/9939ede7-5461-428d-86db-cd66a44cac59-kube-api-access-dpc66\") pod \"cert-manager-55b4d954d5-jkmw9\" (UID: \"9939ede7-5461-428d-86db-cd66a44cac59\") " pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.685346 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.685305    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jkmw9"
Sep 11 03:59:20.965110 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.965087    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jkmw9"]
Sep 11 03:59:20.967275 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:59:20.967240    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9939ede7_5461_428d_86db_cd66a44cac59.slice/crio-d8971e6238e4f37d6e0edb4b595cdb7e522e663f0875a23e493205ec64f6dd21 WatchSource:0}: Error finding container d8971e6238e4f37d6e0edb4b595cdb7e522e663f0875a23e493205ec64f6dd21: Status 404 returned error can't find the container with id d8971e6238e4f37d6e0edb4b595cdb7e522e663f0875a23e493205ec64f6dd21
Sep 11 03:59:20.969311 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:20.969293    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:59:21.835910 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:21.835848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jkmw9" event={"ID":"9939ede7-5461-428d-86db-cd66a44cac59","Type":"ContainerStarted","Data":"d8971e6238e4f37d6e0edb4b595cdb7e522e663f0875a23e493205ec64f6dd21"}
Sep 11 03:59:22.408868 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.408834    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"]
Sep 11 03:59:22.415798 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.415768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:22.418954 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.418808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:59:22.419946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.419541    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-thkch\""
Sep 11 03:59:22.419946 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.419778    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 03:59:22.420996 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.420614    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"]
Sep 11 03:59:22.537764 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.537689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxs9\" (UniqueName: \"kubernetes.io/projected/baa1e1ba-a4ad-450a-b32a-8067cd923703-kube-api-access-ntxs9\") pod \"openshift-lws-operator-fd8ccff4c-5kn6d\" (UID: \"baa1e1ba-a4ad-450a-b32a-8067cd923703\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:22.537764 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.537763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baa1e1ba-a4ad-450a-b32a-8067cd923703-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5kn6d\" (UID: \"baa1e1ba-a4ad-450a-b32a-8067cd923703\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:22.639412 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.638865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxs9\" (UniqueName: \"kubernetes.io/projected/baa1e1ba-a4ad-450a-b32a-8067cd923703-kube-api-access-ntxs9\") pod \"openshift-lws-operator-fd8ccff4c-5kn6d\" (UID: \"baa1e1ba-a4ad-450a-b32a-8067cd923703\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:22.639412 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.638952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baa1e1ba-a4ad-450a-b32a-8067cd923703-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5kn6d\" (UID: \"baa1e1ba-a4ad-450a-b32a-8067cd923703\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:22.639412 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.639363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baa1e1ba-a4ad-450a-b32a-8067cd923703-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5kn6d\" (UID: \"baa1e1ba-a4ad-450a-b32a-8067cd923703\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:22.648418 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.648366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxs9\" (UniqueName: \"kubernetes.io/projected/baa1e1ba-a4ad-450a-b32a-8067cd923703-kube-api-access-ntxs9\") pod \"openshift-lws-operator-fd8ccff4c-5kn6d\" (UID: \"baa1e1ba-a4ad-450a-b32a-8067cd923703\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:22.730351 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:22.730315    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"
Sep 11 03:59:24.150048 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:24.150023    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d"]
Sep 11 03:59:24.151880 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:59:24.151849    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaa1e1ba_a4ad_450a_b32a_8067cd923703.slice/crio-9ade2f9be8b6f3e71c327fa93a32466c29275bc75b4f3e0f6b5b3f0c0efe77e9 WatchSource:0}: Error finding container 9ade2f9be8b6f3e71c327fa93a32466c29275bc75b4f3e0f6b5b3f0c0efe77e9: Status 404 returned error can't find the container with id 9ade2f9be8b6f3e71c327fa93a32466c29275bc75b4f3e0f6b5b3f0c0efe77e9
Sep 11 03:59:24.851623 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:24.851584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d" event={"ID":"baa1e1ba-a4ad-450a-b32a-8067cd923703","Type":"ContainerStarted","Data":"9ade2f9be8b6f3e71c327fa93a32466c29275bc75b4f3e0f6b5b3f0c0efe77e9"}
Sep 11 03:59:24.853060 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:24.853035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jkmw9" event={"ID":"9939ede7-5461-428d-86db-cd66a44cac59","Type":"ContainerStarted","Data":"94c20026aebfcfe09f909360bd5f6344f4ca56cf99bc9db044ec5b090838f04b"}
Sep 11 03:59:24.869570 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:24.869504    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jkmw9" podStartSLOduration=1.8822785720000001 podStartE2EDuration="4.869490414s" totalImagesPullingTime="2.987211842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:24.867843994 +0000 UTC m=+339.132322366" watchObservedRunningTime="2026-09-11 03:59:24.869490414 +0000 UTC m=+339.133968786"
Sep 11 03:59:26.865048 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:26.865010    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d" event={"ID":"baa1e1ba-a4ad-450a-b32a-8067cd923703","Type":"ContainerStarted","Data":"634aaeafe6a9289722e71223d6cb798112ac371980afb81e325f0b2f6caf35da"}
Sep 11 03:59:26.880667 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:26.880617    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5kn6d" podStartSLOduration=2.296754375 podStartE2EDuration="4.880603166s" totalImagesPullingTime="2.583848791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:26.879834003 +0000 UTC m=+341.144312376" watchObservedRunningTime="2026-09-11 03:59:26.880603166 +0000 UTC m=+341.145081537"
Sep 11 03:59:43.354170 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.354095    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"]
Sep 11 03:59:43.357514 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.357494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.360275 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.360242    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 03:59:43.360275 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.360269    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 03:59:43.360442 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.360294    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 03:59:43.360573 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.360538    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 03:59:43.360647 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.360633    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nmbvm\""
Sep 11 03:59:43.385586 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.385539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"]
Sep 11 03:59:43.410865 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.410834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbbebbc1-7a59-4011-8293-b64c404a1bc4-tmp\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.410997 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.410905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbbebbc1-7a59-4011-8293-b64c404a1bc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.410997 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.410961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbbebbc1-7a59-4011-8293-b64c404a1bc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.410997 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.410982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn7n9\" (UniqueName: \"kubernetes.io/projected/fbbebbc1-7a59-4011-8293-b64c404a1bc4-kube-api-access-sn7n9\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.411115 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.411019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbbebbc1-7a59-4011-8293-b64c404a1bc4-manifests\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.512417 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.512383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbbebbc1-7a59-4011-8293-b64c404a1bc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.512417 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.512425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbbebbc1-7a59-4011-8293-b64c404a1bc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.512698 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.512441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn7n9\" (UniqueName: \"kubernetes.io/projected/fbbebbc1-7a59-4011-8293-b64c404a1bc4-kube-api-access-sn7n9\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.512698 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.512469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbbebbc1-7a59-4011-8293-b64c404a1bc4-manifests\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.512698 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.512499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbbebbc1-7a59-4011-8293-b64c404a1bc4-tmp\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.512994 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.512965    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fbbebbc1-7a59-4011-8293-b64c404a1bc4-manifests\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.514730 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.514702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbbebbc1-7a59-4011-8293-b64c404a1bc4-tmp\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.514876 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.514856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fbbebbc1-7a59-4011-8293-b64c404a1bc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.514937 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.514875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbbebbc1-7a59-4011-8293-b64c404a1bc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.524117 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.524091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn7n9\" (UniqueName: \"kubernetes.io/projected/fbbebbc1-7a59-4011-8293-b64c404a1bc4-kube-api-access-sn7n9\") pod \"opendatahub-operator-controller-manager-65b869c898-tptr7\" (UID: \"fbbebbc1-7a59-4011-8293-b64c404a1bc4\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.669729 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.669700    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:43.953996 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:43.953946    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"]
Sep 11 03:59:43.956177 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:59:43.956152    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbebbc1_7a59_4011_8293_b64c404a1bc4.slice/crio-02814ae88d741bba5cdf6ce6e4fd0a7ef029bb7e755ed633caffab73259584c4 WatchSource:0}: Error finding container 02814ae88d741bba5cdf6ce6e4fd0a7ef029bb7e755ed633caffab73259584c4: Status 404 returned error can't find the container with id 02814ae88d741bba5cdf6ce6e4fd0a7ef029bb7e755ed633caffab73259584c4
Sep 11 03:59:44.943722 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:44.943679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7" event={"ID":"fbbebbc1-7a59-4011-8293-b64c404a1bc4","Type":"ContainerStarted","Data":"02814ae88d741bba5cdf6ce6e4fd0a7ef029bb7e755ed633caffab73259584c4"}
Sep 11 03:59:46.954613 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:46.954584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7" event={"ID":"fbbebbc1-7a59-4011-8293-b64c404a1bc4","Type":"ContainerStarted","Data":"66845b54b94e8249ba5698f9bdc7d02cf4c31b8910323023520336c28790b39c"}
Sep 11 03:59:48.966488 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:48.966456    2582 generic.go:356] "Generic (PLEG): container finished" podID="fbbebbc1-7a59-4011-8293-b64c404a1bc4" containerID="66845b54b94e8249ba5698f9bdc7d02cf4c31b8910323023520336c28790b39c" exitCode=0
Sep 11 03:59:48.966882 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:48.966540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7" event={"ID":"fbbebbc1-7a59-4011-8293-b64c404a1bc4","Type":"ContainerDied","Data":"66845b54b94e8249ba5698f9bdc7d02cf4c31b8910323023520336c28790b39c"}
Sep 11 03:59:49.973416 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:49.973382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7" event={"ID":"fbbebbc1-7a59-4011-8293-b64c404a1bc4","Type":"ContainerStarted","Data":"ac2a118bb20c94943d6bde079966f608dff772c96a5c698907b2f648763b8d38"}
Sep 11 03:59:49.973848 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:49.973608    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 03:59:49.996568 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:49.996507    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7" podStartSLOduration=2.59405216 podStartE2EDuration="6.996492924s" totalImagesPullingTime="3.357403878s" totalInitContainerRuntime="1.045036886s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:49.994477585 +0000 UTC m=+364.258955959" watchObservedRunningTime="2026-09-11 03:59:49.996492924 +0000 UTC m=+364.260971295"
Sep 11 03:59:52.549242 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.549212    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dbb545bc8-8sdrq"]
Sep 11 03:59:52.558114 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.558089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.574577 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.574245    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbb545bc8-8sdrq"]
Sep 11 03:59:52.594378 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.594341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-serving-cert\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.594541 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.594424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-config\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.594541 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.594485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-service-ca\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.594541 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.594506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-trusted-ca-bundle\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.594714 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.594619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-oauth-config\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.594714 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.594652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-oauth-serving-cert\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.594714 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.594668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbxl\" (UniqueName: \"kubernetes.io/projected/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-kube-api-access-tsbxl\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.695262 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.695225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-config\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.695440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.695276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-service-ca\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.695440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.695292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-trusted-ca-bundle\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.695440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.695323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-oauth-config\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.695440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.695340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-oauth-serving-cert\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.695440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.695379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbxl\" (UniqueName: \"kubernetes.io/projected/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-kube-api-access-tsbxl\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.695440 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.695418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-serving-cert\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.696102 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.696073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-service-ca\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.696231 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.696153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-trusted-ca-bundle\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.696386 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.696362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-oauth-serving-cert\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.696534 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.696518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-config\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.698320 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.698297    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-serving-cert\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.698433 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.698400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-console-oauth-config\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.705784 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.705763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbxl\" (UniqueName: \"kubernetes.io/projected/59e4ecae-b7e0-4734-80a2-3bd4795a9c79-kube-api-access-tsbxl\") pod \"console-5dbb545bc8-8sdrq\" (UID: \"59e4ecae-b7e0-4734-80a2-3bd4795a9c79\") " pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:52.867284 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:52.867196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 03:59:53.154169 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:53.154098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbb545bc8-8sdrq"]
Sep 11 03:59:53.157614 ip-10-0-138-179 kubenswrapper[2582]: W0911 03:59:53.157584    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e4ecae_b7e0_4734_80a2_3bd4795a9c79.slice/crio-2ec45e0b3b26c4c08e9a1ce26d8194292529ddfb1ff8b7c8bb6bf3e7da118c89 WatchSource:0}: Error finding container 2ec45e0b3b26c4c08e9a1ce26d8194292529ddfb1ff8b7c8bb6bf3e7da118c89: Status 404 returned error can't find the container with id 2ec45e0b3b26c4c08e9a1ce26d8194292529ddfb1ff8b7c8bb6bf3e7da118c89
Sep 11 03:59:53.993858 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:53.993825    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbb545bc8-8sdrq" event={"ID":"59e4ecae-b7e0-4734-80a2-3bd4795a9c79","Type":"ContainerStarted","Data":"755d9838adf8a3d7340dcdf41ebc3f823e8e171d999a497ab5b8ee617d28fc10"}
Sep 11 03:59:53.994323 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:53.993881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbb545bc8-8sdrq" event={"ID":"59e4ecae-b7e0-4734-80a2-3bd4795a9c79","Type":"ContainerStarted","Data":"2ec45e0b3b26c4c08e9a1ce26d8194292529ddfb1ff8b7c8bb6bf3e7da118c89"}
Sep 11 03:59:54.029494 ip-10-0-138-179 kubenswrapper[2582]: I0911 03:59:54.029442    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dbb545bc8-8sdrq" podStartSLOduration=2.029425534 podStartE2EDuration="2.029425534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:54.027540591 +0000 UTC m=+368.292018961" watchObservedRunningTime="2026-09-11 03:59:54.029425534 +0000 UTC m=+368.293903905"
Sep 11 04:00:00.979951 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:00.979914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-tptr7"
Sep 11 04:00:02.868127 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:02.868093    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 04:00:02.868501 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:02.868153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 04:00:02.872738 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:02.872709    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 04:00:03.039122 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.039095    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dbb545bc8-8sdrq"
Sep 11 04:00:03.213387 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.213351    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"]
Sep 11 04:00:03.219679 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.219659    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.229427 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.229401    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 04:00:03.229575 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.229502    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 04:00:03.229758 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.229744    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 04:00:03.230016 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.229979    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-t7hw4\""
Sep 11 04:00:03.251321 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.251296    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-589764645d-cc26k"]
Sep 11 04:00:03.274401 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.274371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"]
Sep 11 04:00:03.292892 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.292868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dd94aeb-2aeb-4ee5-95a4-90601016d763-cert\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.293011 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.292903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlp6n\" (UniqueName: \"kubernetes.io/projected/4dd94aeb-2aeb-4ee5-95a4-90601016d763-kube-api-access-nlp6n\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.293011 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.292938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4dd94aeb-2aeb-4ee5-95a4-90601016d763-manager-config\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.293011 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.292959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dd94aeb-2aeb-4ee5-95a4-90601016d763-metrics-cert\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.393884 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.393848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dd94aeb-2aeb-4ee5-95a4-90601016d763-cert\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.394048 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.393890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlp6n\" (UniqueName: \"kubernetes.io/projected/4dd94aeb-2aeb-4ee5-95a4-90601016d763-kube-api-access-nlp6n\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.394048 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.393927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4dd94aeb-2aeb-4ee5-95a4-90601016d763-manager-config\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.394048 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.393950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dd94aeb-2aeb-4ee5-95a4-90601016d763-metrics-cert\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.394699 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.394674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4dd94aeb-2aeb-4ee5-95a4-90601016d763-manager-config\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.396300 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.396275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dd94aeb-2aeb-4ee5-95a4-90601016d763-cert\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.396387 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.396310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dd94aeb-2aeb-4ee5-95a4-90601016d763-metrics-cert\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.430577 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.430538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlp6n\" (UniqueName: \"kubernetes.io/projected/4dd94aeb-2aeb-4ee5-95a4-90601016d763-kube-api-access-nlp6n\") pod \"lws-controller-manager-5f8f45d65c-rxm5h\" (UID: \"4dd94aeb-2aeb-4ee5-95a4-90601016d763\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.530032 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.529944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:03.544215 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.544175    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"]
Sep 11 04:00:03.546842 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.546815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.549712 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.549687    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 04:00:03.549825 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.549728    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 04:00:03.550237 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.550199    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 04:00:03.550363 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.550263    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 04:00:03.550505 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.550489    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pn9zb\""
Sep 11 04:00:03.571078 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.571028    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"]
Sep 11 04:00:03.596382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.596343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3675281-98a6-412e-8ab3-aa70b23d52fe-tls-certs\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.596546 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.596404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3675281-98a6-412e-8ab3-aa70b23d52fe-tmp\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.596546 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.596450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfh2\" (UniqueName: \"kubernetes.io/projected/d3675281-98a6-412e-8ab3-aa70b23d52fe-kube-api-access-8mfh2\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.697935 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.697894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3675281-98a6-412e-8ab3-aa70b23d52fe-tls-certs\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.697935 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.697938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3675281-98a6-412e-8ab3-aa70b23d52fe-tmp\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.698134 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.697976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfh2\" (UniqueName: \"kubernetes.io/projected/d3675281-98a6-412e-8ab3-aa70b23d52fe-kube-api-access-8mfh2\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.700284 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.700258    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3675281-98a6-412e-8ab3-aa70b23d52fe-tmp\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.700420 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.700403    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3675281-98a6-412e-8ab3-aa70b23d52fe-tls-certs\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.706058 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.706040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfh2\" (UniqueName: \"kubernetes.io/projected/d3675281-98a6-412e-8ab3-aa70b23d52fe-kube-api-access-8mfh2\") pod \"kube-auth-proxy-7f78fb9f44-7c4gt\" (UID: \"d3675281-98a6-412e-8ab3-aa70b23d52fe\") " pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:03.809947 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.809921    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"]
Sep 11 04:00:03.812426 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:00:03.812404    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd94aeb_2aeb_4ee5_95a4_90601016d763.slice/crio-103d7ef35f7cc6c03eb361ba9ea005ffa08db17bee52b0bb9f70f6729492b6bd WatchSource:0}: Error finding container 103d7ef35f7cc6c03eb361ba9ea005ffa08db17bee52b0bb9f70f6729492b6bd: Status 404 returned error can't find the container with id 103d7ef35f7cc6c03eb361ba9ea005ffa08db17bee52b0bb9f70f6729492b6bd
Sep 11 04:00:03.859641 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:03.859607    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"
Sep 11 04:00:04.040851 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:04.040813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h" event={"ID":"4dd94aeb-2aeb-4ee5-95a4-90601016d763","Type":"ContainerStarted","Data":"103d7ef35f7cc6c03eb361ba9ea005ffa08db17bee52b0bb9f70f6729492b6bd"}
Sep 11 04:00:04.134144 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:04.134116    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt"]
Sep 11 04:00:04.136675 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:00:04.136644    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3675281_98a6_412e_8ab3_aa70b23d52fe.slice/crio-ab28312037a7ea0d833f3ca120c3c6d2eaeeedb2e9f964a9e1830abde8462dea WatchSource:0}: Error finding container ab28312037a7ea0d833f3ca120c3c6d2eaeeedb2e9f964a9e1830abde8462dea: Status 404 returned error can't find the container with id ab28312037a7ea0d833f3ca120c3c6d2eaeeedb2e9f964a9e1830abde8462dea
Sep 11 04:00:05.046496 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:05.046458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt" event={"ID":"d3675281-98a6-412e-8ab3-aa70b23d52fe","Type":"ContainerStarted","Data":"ab28312037a7ea0d833f3ca120c3c6d2eaeeedb2e9f964a9e1830abde8462dea"}
Sep 11 04:00:08.065918 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:08.065875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt" event={"ID":"d3675281-98a6-412e-8ab3-aa70b23d52fe","Type":"ContainerStarted","Data":"adcaff711fb4bccba4d252b033740487f1f3ccf5ff64f672c5d3a95b8cd56974"}
Sep 11 04:00:08.081610 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:08.081517    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f78fb9f44-7c4gt" podStartSLOduration=1.788081736 podStartE2EDuration="5.081501322s" totalImagesPullingTime="3.293419586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:08.080918003 +0000 UTC m=+382.345396375" watchObservedRunningTime="2026-09-11 04:00:08.081501322 +0000 UTC m=+382.345979693"
Sep 11 04:00:09.072023 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:09.071984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h" event={"ID":"4dd94aeb-2aeb-4ee5-95a4-90601016d763","Type":"ContainerStarted","Data":"0c902517bc3dbaf626dfeaa93dec879f2b61340f0b0cadc0236031975145d061"}
Sep 11 04:00:09.072453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:09.072206    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:09.090399 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:09.090346    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h" podStartSLOduration=1.679057295 podStartE2EDuration="6.090332316s" totalImagesPullingTime="4.411275021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:09.088872791 +0000 UTC m=+383.353351162" watchObservedRunningTime="2026-09-11 04:00:09.090332316 +0000 UTC m=+383.354810686"
Sep 11 04:00:20.079228 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:20.079193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-rxm5h"
Sep 11 04:00:28.270087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.270042    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-589764645d-cc26k" podUID="8d017451-ced0-4fe7-9332-0d6a725af531" containerName="console" containerID="cri-o://1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e" gracePeriod=15
Sep 11 04:00:28.515155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.515126    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589764645d-cc26k"
Sep 11 04:00:28.522586 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522522    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-console-config\") pod \"8d017451-ced0-4fe7-9332-0d6a725af531\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") "
Sep 11 04:00:28.522586 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522552    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-oauth-config\") pod \"8d017451-ced0-4fe7-9332-0d6a725af531\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") "
Sep 11 04:00:28.522721 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-service-ca\") pod \"8d017451-ced0-4fe7-9332-0d6a725af531\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") "
Sep 11 04:00:28.522721 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-trusted-ca-bundle\") pod \"8d017451-ced0-4fe7-9332-0d6a725af531\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") "
Sep 11 04:00:28.522721 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522699    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-oauth-serving-cert\") pod \"8d017451-ced0-4fe7-9332-0d6a725af531\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") "
Sep 11 04:00:28.523053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522735    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-serving-cert\") pod \"8d017451-ced0-4fe7-9332-0d6a725af531\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") "
Sep 11 04:00:28.523053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522784    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk8cw\" (UniqueName: \"kubernetes.io/projected/8d017451-ced0-4fe7-9332-0d6a725af531-kube-api-access-jk8cw\") pod \"8d017451-ced0-4fe7-9332-0d6a725af531\" (UID: \"8d017451-ced0-4fe7-9332-0d6a725af531\") "
Sep 11 04:00:28.523053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-console-config" (OuterVolumeSpecName: "console-config") pod "8d017451-ced0-4fe7-9332-0d6a725af531" (UID: "8d017451-ced0-4fe7-9332-0d6a725af531"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:00:28.523053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.522957    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d017451-ced0-4fe7-9332-0d6a725af531" (UID: "8d017451-ced0-4fe7-9332-0d6a725af531"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:00:28.523265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.523114    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-console-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:00:28.523265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.523127    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:00:28.523265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.523121    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8d017451-ced0-4fe7-9332-0d6a725af531" (UID: "8d017451-ced0-4fe7-9332-0d6a725af531"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:00:28.523265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.523129    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d017451-ced0-4fe7-9332-0d6a725af531" (UID: "8d017451-ced0-4fe7-9332-0d6a725af531"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:00:28.524810 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.524787    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d017451-ced0-4fe7-9332-0d6a725af531" (UID: "8d017451-ced0-4fe7-9332-0d6a725af531"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:00:28.524926 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.524899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d017451-ced0-4fe7-9332-0d6a725af531-kube-api-access-jk8cw" (OuterVolumeSpecName: "kube-api-access-jk8cw") pod "8d017451-ced0-4fe7-9332-0d6a725af531" (UID: "8d017451-ced0-4fe7-9332-0d6a725af531"). InnerVolumeSpecName "kube-api-access-jk8cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:00:28.524976 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.524926    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d017451-ced0-4fe7-9332-0d6a725af531" (UID: "8d017451-ced0-4fe7-9332-0d6a725af531"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:00:28.624195 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.624166    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-oauth-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:00:28.624195 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.624192    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-serving-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:00:28.624195 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.624203    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8cw\" (UniqueName: \"kubernetes.io/projected/8d017451-ced0-4fe7-9332-0d6a725af531-kube-api-access-jk8cw\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:00:28.624417 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.624211    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d017451-ced0-4fe7-9332-0d6a725af531-console-oauth-config\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:00:28.624417 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:28.624220    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d017451-ced0-4fe7-9332-0d6a725af531-trusted-ca-bundle\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:00:29.166353 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.166319    2582 generic.go:356] "Generic (PLEG): container finished" podID="8d017451-ced0-4fe7-9332-0d6a725af531" containerID="1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e" exitCode=0
Sep 11 04:00:29.166353 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.166356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589764645d-cc26k" event={"ID":"8d017451-ced0-4fe7-9332-0d6a725af531","Type":"ContainerDied","Data":"1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e"}
Sep 11 04:00:29.166627 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.166361    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-589764645d-cc26k"
Sep 11 04:00:29.166627 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.166396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-589764645d-cc26k" event={"ID":"8d017451-ced0-4fe7-9332-0d6a725af531","Type":"ContainerDied","Data":"6a980671fd243f17d2c734a8ed86d8ba961ab545d7c3f088f7d1ca58743fb025"}
Sep 11 04:00:29.166627 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.166420    2582 scope.go:117] "RemoveContainer" containerID="1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e"
Sep 11 04:00:29.175769 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.175755    2582 scope.go:117] "RemoveContainer" containerID="1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e"
Sep 11 04:00:29.176031 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:00:29.176014    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e\": container with ID starting with 1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e not found: ID does not exist" containerID="1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e"
Sep 11 04:00:29.176085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.176041    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e"} err="failed to get container status \"1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e\": rpc error: code = NotFound desc = could not find container \"1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e\": container with ID starting with 1e9852c2c322480b1f90301e985f12373268f42f2d1365b73634c222f377919e not found: ID does not exist"
Sep 11 04:00:29.189366 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.189334    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-589764645d-cc26k"]
Sep 11 04:00:29.193504 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:29.193484    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-589764645d-cc26k"]
Sep 11 04:00:30.273789 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:00:30.273759    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d017451-ced0-4fe7-9332-0d6a725af531" path="/var/lib/kubelet/pods/8d017451-ced0-4fe7-9332-0d6a725af531/volumes"
Sep 11 04:01:41.381419 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.381384    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hmz2j"]
Sep 11 04:01:41.381890 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.381767    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d017451-ced0-4fe7-9332-0d6a725af531" containerName="console"
Sep 11 04:01:41.381890 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.381782    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d017451-ced0-4fe7-9332-0d6a725af531" containerName="console"
Sep 11 04:01:41.381890 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.381853    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d017451-ced0-4fe7-9332-0d6a725af531" containerName="console"
Sep 11 04:01:41.384827 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.384808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j"
Sep 11 04:01:41.389034 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.389011    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 04:01:41.389155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.389056    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rdm2x\""
Sep 11 04:01:41.389220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.389202    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 04:01:41.397987 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.397965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hmz2j"]
Sep 11 04:01:41.404921 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.404896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rh4r\" (UniqueName: \"kubernetes.io/projected/ba001e73-cea6-4982-9e26-d42428ffd640-kube-api-access-9rh4r\") pod \"authorino-operator-76b478bd78-hmz2j\" (UID: \"ba001e73-cea6-4982-9e26-d42428ffd640\") " pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j"
Sep 11 04:01:41.505799 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.505763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rh4r\" (UniqueName: \"kubernetes.io/projected/ba001e73-cea6-4982-9e26-d42428ffd640-kube-api-access-9rh4r\") pod \"authorino-operator-76b478bd78-hmz2j\" (UID: \"ba001e73-cea6-4982-9e26-d42428ffd640\") " pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j"
Sep 11 04:01:41.514167 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.514137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rh4r\" (UniqueName: \"kubernetes.io/projected/ba001e73-cea6-4982-9e26-d42428ffd640-kube-api-access-9rh4r\") pod \"authorino-operator-76b478bd78-hmz2j\" (UID: \"ba001e73-cea6-4982-9e26-d42428ffd640\") " pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j"
Sep 11 04:01:41.694968 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.694943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j"
Sep 11 04:01:41.966329 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:41.966255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hmz2j"]
Sep 11 04:01:41.969957 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:01:41.969928    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba001e73_cea6_4982_9e26_d42428ffd640.slice/crio-3ac370dc3db453724c6827b501c7bdbc60f14027d5b9bb650605eefc82c860ca WatchSource:0}: Error finding container 3ac370dc3db453724c6827b501c7bdbc60f14027d5b9bb650605eefc82c860ca: Status 404 returned error can't find the container with id 3ac370dc3db453724c6827b501c7bdbc60f14027d5b9bb650605eefc82c860ca
Sep 11 04:01:42.507041 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:42.507000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j" event={"ID":"ba001e73-cea6-4982-9e26-d42428ffd640","Type":"ContainerStarted","Data":"3ac370dc3db453724c6827b501c7bdbc60f14027d5b9bb650605eefc82c860ca"}
Sep 11 04:01:44.519778 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:44.519742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j" event={"ID":"ba001e73-cea6-4982-9e26-d42428ffd640","Type":"ContainerStarted","Data":"501338814bfd06d1e428785d1c48fa1817e85cf2acabea84782086504ae8a0d2"}
Sep 11 04:01:44.520257 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:44.519865    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j"
Sep 11 04:01:44.538993 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:44.538943    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j" podStartSLOduration=1.399084701 podStartE2EDuration="3.538928371s" totalImagesPullingTime="2.13984367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:44.536884074 +0000 UTC m=+478.801362445" watchObservedRunningTime="2026-09-11 04:01:44.538928371 +0000 UTC m=+478.803406742"
Sep 11 04:01:55.527217 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:01:55.527186    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-hmz2j"
Sep 11 04:02:41.952266 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:41.952191    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8rtpt"]
Sep 11 04:02:41.955349 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:41.955332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt"
Sep 11 04:02:41.957862 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:41.957840    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rmb4f\""
Sep 11 04:02:41.962583 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:41.962544    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8rtpt"]
Sep 11 04:02:42.057921 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:42.057885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6sf\" (UniqueName: \"kubernetes.io/projected/3206cdad-0d64-492e-8096-1252476704ad-kube-api-access-9r6sf\") pod \"authorino-7bcfd8f9cd-8rtpt\" (UID: \"3206cdad-0d64-492e-8096-1252476704ad\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt"
Sep 11 04:02:42.158784 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:42.158745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6sf\" (UniqueName: \"kubernetes.io/projected/3206cdad-0d64-492e-8096-1252476704ad-kube-api-access-9r6sf\") pod \"authorino-7bcfd8f9cd-8rtpt\" (UID: \"3206cdad-0d64-492e-8096-1252476704ad\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt"
Sep 11 04:02:42.177981 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:42.177954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6sf\" (UniqueName: \"kubernetes.io/projected/3206cdad-0d64-492e-8096-1252476704ad-kube-api-access-9r6sf\") pod \"authorino-7bcfd8f9cd-8rtpt\" (UID: \"3206cdad-0d64-492e-8096-1252476704ad\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt"
Sep 11 04:02:42.265016 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:42.264942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt"
Sep 11 04:02:42.533444 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:42.533368    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8rtpt"]
Sep 11 04:02:42.536516 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:02:42.536491    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3206cdad_0d64_492e_8096_1252476704ad.slice/crio-4d4954af11a66158f96320c76e6af3bf814fb3063077c722cd1edcbc0e7ac74d WatchSource:0}: Error finding container 4d4954af11a66158f96320c76e6af3bf814fb3063077c722cd1edcbc0e7ac74d: Status 404 returned error can't find the container with id 4d4954af11a66158f96320c76e6af3bf814fb3063077c722cd1edcbc0e7ac74d
Sep 11 04:02:42.791850 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:42.791771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt" event={"ID":"3206cdad-0d64-492e-8096-1252476704ad","Type":"ContainerStarted","Data":"4d4954af11a66158f96320c76e6af3bf814fb3063077c722cd1edcbc0e7ac74d"}
Sep 11 04:02:44.804056 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:44.804015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt" event={"ID":"3206cdad-0d64-492e-8096-1252476704ad","Type":"ContainerStarted","Data":"62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1"}
Sep 11 04:02:44.819978 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:44.819932    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt" podStartSLOduration=2.426447594 podStartE2EDuration="3.819916177s" totalImagesPullingTime="1.393468583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:44.819313954 +0000 UTC m=+539.083792328" watchObservedRunningTime="2026-09-11 04:02:44.819916177 +0000 UTC m=+539.084394798"
Sep 11 04:02:45.151972 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:45.151888    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8rtpt"]
Sep 11 04:02:46.812642 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:46.812600    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt" podUID="3206cdad-0d64-492e-8096-1252476704ad" containerName="authorino" containerID="cri-o://62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1" gracePeriod=30
Sep 11 04:02:47.050600 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.050576    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt"
Sep 11 04:02:47.208314 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.208276    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r6sf\" (UniqueName: \"kubernetes.io/projected/3206cdad-0d64-492e-8096-1252476704ad-kube-api-access-9r6sf\") pod \"3206cdad-0d64-492e-8096-1252476704ad\" (UID: \"3206cdad-0d64-492e-8096-1252476704ad\") "
Sep 11 04:02:47.210462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.210432    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3206cdad-0d64-492e-8096-1252476704ad-kube-api-access-9r6sf" (OuterVolumeSpecName: "kube-api-access-9r6sf") pod "3206cdad-0d64-492e-8096-1252476704ad" (UID: "3206cdad-0d64-492e-8096-1252476704ad"). InnerVolumeSpecName "kube-api-access-9r6sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:02:47.309511 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.309474    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r6sf\" (UniqueName: \"kubernetes.io/projected/3206cdad-0d64-492e-8096-1252476704ad-kube-api-access-9r6sf\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:02:47.818491 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.818458    2582 generic.go:356] "Generic (PLEG): container finished" podID="3206cdad-0d64-492e-8096-1252476704ad" containerID="62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1" exitCode=0
Sep 11 04:02:47.818906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.818498    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt"
Sep 11 04:02:47.818906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.818541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt" event={"ID":"3206cdad-0d64-492e-8096-1252476704ad","Type":"ContainerDied","Data":"62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1"}
Sep 11 04:02:47.818906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.818596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8rtpt" event={"ID":"3206cdad-0d64-492e-8096-1252476704ad","Type":"ContainerDied","Data":"4d4954af11a66158f96320c76e6af3bf814fb3063077c722cd1edcbc0e7ac74d"}
Sep 11 04:02:47.818906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.818613    2582 scope.go:117] "RemoveContainer" containerID="62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1"
Sep 11 04:02:47.827090 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.827070    2582 scope.go:117] "RemoveContainer" containerID="62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1"
Sep 11 04:02:47.827343 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:02:47.827327    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1\": container with ID starting with 62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1 not found: ID does not exist" containerID="62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1"
Sep 11 04:02:47.827392 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.827352    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1"} err="failed to get container status \"62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1\": rpc error: code = NotFound desc = could not find container \"62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1\": container with ID starting with 62c5e621c546aca5ca05d23148f20d572dd4c9bbf7742c33fa9aac07756bc6a1 not found: ID does not exist"
Sep 11 04:02:47.839631 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.839606    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8rtpt"]
Sep 11 04:02:47.843562 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:47.843534    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8rtpt"]
Sep 11 04:02:48.274954 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:02:48.274925    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3206cdad-0d64-492e-8096-1252476704ad" path="/var/lib/kubelet/pods/3206cdad-0d64-492e-8096-1252476704ad/volumes"
Sep 11 04:03:09.047800 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.047759    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-q9wbl"]
Sep 11 04:03:09.048179 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.048166    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3206cdad-0d64-492e-8096-1252476704ad" containerName="authorino"
Sep 11 04:03:09.048222 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.048181    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3206cdad-0d64-492e-8096-1252476704ad" containerName="authorino"
Sep 11 04:03:09.048285 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.048274    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3206cdad-0d64-492e-8096-1252476704ad" containerName="authorino"
Sep 11 04:03:09.052817 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.052798    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-q9wbl"
Sep 11 04:03:09.055536 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.055517    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rmb4f\""
Sep 11 04:03:09.056272 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.056248    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-q9wbl"]
Sep 11 04:03:09.099674 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.099642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7lb\" (UniqueName: \"kubernetes.io/projected/9d0e7c5b-65fa-4e73-94be-933ec753eec5-kube-api-access-df7lb\") pod \"authorino-557f876f69-q9wbl\" (UID: \"9d0e7c5b-65fa-4e73-94be-933ec753eec5\") " pod="kuadrant-system/authorino-557f876f69-q9wbl"
Sep 11 04:03:09.151036 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.151001    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-q9wbl"]
Sep 11 04:03:09.151312 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:03:09.151287    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-df7lb], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-q9wbl" podUID="9d0e7c5b-65fa-4e73-94be-933ec753eec5"
Sep 11 04:03:09.181426 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.181393    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gk2gc"]
Sep 11 04:03:09.184873 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.184851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.187326 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.187306    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 04:03:09.191657 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.191628    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gk2gc"]
Sep 11 04:03:09.200914 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.200890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7lb\" (UniqueName: \"kubernetes.io/projected/9d0e7c5b-65fa-4e73-94be-933ec753eec5-kube-api-access-df7lb\") pod \"authorino-557f876f69-q9wbl\" (UID: \"9d0e7c5b-65fa-4e73-94be-933ec753eec5\") " pod="kuadrant-system/authorino-557f876f69-q9wbl"
Sep 11 04:03:09.201031 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.200935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96ac5a5-8750-484e-b17c-06c86310f1d0-tls-cert\") pod \"authorino-6d4dfc649f-gk2gc\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") " pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.201031 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.200967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9g9\" (UniqueName: \"kubernetes.io/projected/d96ac5a5-8750-484e-b17c-06c86310f1d0-kube-api-access-pp9g9\") pod \"authorino-6d4dfc649f-gk2gc\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") " pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.208952 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.208922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7lb\" (UniqueName: \"kubernetes.io/projected/9d0e7c5b-65fa-4e73-94be-933ec753eec5-kube-api-access-df7lb\") pod \"authorino-557f876f69-q9wbl\" (UID: \"9d0e7c5b-65fa-4e73-94be-933ec753eec5\") " pod="kuadrant-system/authorino-557f876f69-q9wbl"
Sep 11 04:03:09.302459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.302351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96ac5a5-8750-484e-b17c-06c86310f1d0-tls-cert\") pod \"authorino-6d4dfc649f-gk2gc\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") " pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.302924 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.302598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9g9\" (UniqueName: \"kubernetes.io/projected/d96ac5a5-8750-484e-b17c-06c86310f1d0-kube-api-access-pp9g9\") pod \"authorino-6d4dfc649f-gk2gc\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") " pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.304733 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.304705    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gk2gc"]
Sep 11 04:03:09.304938 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:03:09.304919    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pp9g9 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-gk2gc" podUID="d96ac5a5-8750-484e-b17c-06c86310f1d0"
Sep 11 04:03:09.305344 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.305321    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96ac5a5-8750-484e-b17c-06c86310f1d0-tls-cert\") pod \"authorino-6d4dfc649f-gk2gc\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") " pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.313287 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.313255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9g9\" (UniqueName: \"kubernetes.io/projected/d96ac5a5-8750-484e-b17c-06c86310f1d0-kube-api-access-pp9g9\") pod \"authorino-6d4dfc649f-gk2gc\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") " pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.332740 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.332708    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58c69d57bf-cvdpl"]
Sep 11 04:03:09.336226 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.336202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.342865 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.342836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58c69d57bf-cvdpl"]
Sep 11 04:03:09.404185 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.404144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58pd\" (UniqueName: \"kubernetes.io/projected/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-kube-api-access-h58pd\") pod \"authorino-58c69d57bf-cvdpl\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") " pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.404378 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.404285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-tls-cert\") pod \"authorino-58c69d57bf-cvdpl\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") " pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.504817 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.504779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-tls-cert\") pod \"authorino-58c69d57bf-cvdpl\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") " pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.505144 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.504827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58pd\" (UniqueName: \"kubernetes.io/projected/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-kube-api-access-h58pd\") pod \"authorino-58c69d57bf-cvdpl\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") " pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.507373 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.507351    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-tls-cert\") pod \"authorino-58c69d57bf-cvdpl\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") " pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.512769 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.512749    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58pd\" (UniqueName: \"kubernetes.io/projected/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-kube-api-access-h58pd\") pod \"authorino-58c69d57bf-cvdpl\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") " pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.647184 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.647083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:03:09.916373 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.916346    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58c69d57bf-cvdpl"]
Sep 11 04:03:09.918870 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:03:09.918844    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cf3ac5e_4889_4363_832c_1c5ef5e8de92.slice/crio-d68539ef16bec560132ac6ea29d17ad1f2a8fce9c47afb3fe74fffea555f3113 WatchSource:0}: Error finding container d68539ef16bec560132ac6ea29d17ad1f2a8fce9c47afb3fe74fffea555f3113: Status 404 returned error can't find the container with id d68539ef16bec560132ac6ea29d17ad1f2a8fce9c47afb3fe74fffea555f3113
Sep 11 04:03:09.926592 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.926564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c69d57bf-cvdpl" event={"ID":"5cf3ac5e-4889-4363-832c-1c5ef5e8de92","Type":"ContainerStarted","Data":"d68539ef16bec560132ac6ea29d17ad1f2a8fce9c47afb3fe74fffea555f3113"}
Sep 11 04:03:09.926688 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.926592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.926688 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.926591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-q9wbl"
Sep 11 04:03:09.933873 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.933855    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:09.936697 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:09.936680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-q9wbl"
Sep 11 04:03:10.009112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.009085    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df7lb\" (UniqueName: \"kubernetes.io/projected/9d0e7c5b-65fa-4e73-94be-933ec753eec5-kube-api-access-df7lb\") pod \"9d0e7c5b-65fa-4e73-94be-933ec753eec5\" (UID: \"9d0e7c5b-65fa-4e73-94be-933ec753eec5\") "
Sep 11 04:03:10.009254 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.009142    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp9g9\" (UniqueName: \"kubernetes.io/projected/d96ac5a5-8750-484e-b17c-06c86310f1d0-kube-api-access-pp9g9\") pod \"d96ac5a5-8750-484e-b17c-06c86310f1d0\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") "
Sep 11 04:03:10.009254 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.009189    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96ac5a5-8750-484e-b17c-06c86310f1d0-tls-cert\") pod \"d96ac5a5-8750-484e-b17c-06c86310f1d0\" (UID: \"d96ac5a5-8750-484e-b17c-06c86310f1d0\") "
Sep 11 04:03:10.011161 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.011134    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0e7c5b-65fa-4e73-94be-933ec753eec5-kube-api-access-df7lb" (OuterVolumeSpecName: "kube-api-access-df7lb") pod "9d0e7c5b-65fa-4e73-94be-933ec753eec5" (UID: "9d0e7c5b-65fa-4e73-94be-933ec753eec5"). InnerVolumeSpecName "kube-api-access-df7lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:10.011267 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.011176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96ac5a5-8750-484e-b17c-06c86310f1d0-kube-api-access-pp9g9" (OuterVolumeSpecName: "kube-api-access-pp9g9") pod "d96ac5a5-8750-484e-b17c-06c86310f1d0" (UID: "d96ac5a5-8750-484e-b17c-06c86310f1d0"). InnerVolumeSpecName "kube-api-access-pp9g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:10.011267 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.011199    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96ac5a5-8750-484e-b17c-06c86310f1d0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d96ac5a5-8750-484e-b17c-06c86310f1d0" (UID: "d96ac5a5-8750-484e-b17c-06c86310f1d0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:10.110754 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.110725    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df7lb\" (UniqueName: \"kubernetes.io/projected/9d0e7c5b-65fa-4e73-94be-933ec753eec5-kube-api-access-df7lb\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:03:10.110754 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.110750    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp9g9\" (UniqueName: \"kubernetes.io/projected/d96ac5a5-8750-484e-b17c-06c86310f1d0-kube-api-access-pp9g9\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:03:10.110754 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.110759    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96ac5a5-8750-484e-b17c-06c86310f1d0-tls-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:03:10.932979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.932942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c69d57bf-cvdpl" event={"ID":"5cf3ac5e-4889-4363-832c-1c5ef5e8de92","Type":"ContainerStarted","Data":"86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a"}
Sep 11 04:03:10.933191 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.933052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gk2gc"
Sep 11 04:03:10.933191 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.933129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-q9wbl"
Sep 11 04:03:10.948883 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.948839    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58c69d57bf-cvdpl" podStartSLOduration=1.377932006 podStartE2EDuration="1.948823417s" totalImagesPullingTime="570.891411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:10.947483741 +0000 UTC m=+565.211962112" watchObservedRunningTime="2026-09-11 04:03:10.948823417 +0000 UTC m=+565.213301836"
Sep 11 04:03:10.971774 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.971732    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gk2gc"]
Sep 11 04:03:10.981546 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:10.981519    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gk2gc"]
Sep 11 04:03:11.002710 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:11.002678    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-q9wbl"]
Sep 11 04:03:11.006365 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:11.006341    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-q9wbl"]
Sep 11 04:03:12.273740 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:12.273710    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0e7c5b-65fa-4e73-94be-933ec753eec5" path="/var/lib/kubelet/pods/9d0e7c5b-65fa-4e73-94be-933ec753eec5/volumes"
Sep 11 04:03:12.274139 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:12.273994    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96ac5a5-8750-484e-b17c-06c86310f1d0" path="/var/lib/kubelet/pods/d96ac5a5-8750-484e-b17c-06c86310f1d0/volumes"
Sep 11 04:03:16.016198 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.016165    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d9bd4c4c-k8nr2"]
Sep 11 04:03:16.019945 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.019924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.022295 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.022269    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 04:03:16.022426 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.022303    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 04:03:16.022426 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.022364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 04:03:16.022426 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.022394    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qwhqf\""
Sep 11 04:03:16.030939 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.030915    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9bd4c4c-k8nr2"]
Sep 11 04:03:16.062604 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.062546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.062604 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.062610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2ss\" (UniqueName: \"kubernetes.io/projected/831a5ce5-5868-4474-975e-0d678897c578-kube-api-access-6g2ss\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.062812 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.062635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.163188 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.163154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.163188 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.163194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2ss\" (UniqueName: \"kubernetes.io/projected/831a5ce5-5868-4474-975e-0d678897c578-kube-api-access-6g2ss\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.163383 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.163219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.163383 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:03:16.163306    2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 04:03:16.163383 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:03:16.163346    2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 04:03:16.163484 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:03:16.163361    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert podName:831a5ce5-5868-4474-975e-0d678897c578 nodeName:}" failed. No retries permitted until 2026-09-11 04:03:16.663344235 +0000 UTC m=+570.927822583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert") pod "maas-controller-5d9bd4c4c-k8nr2" (UID: "831a5ce5-5868-4474-975e-0d678897c578") : secret "maas-controller-webhook-cert" not found
Sep 11 04:03:16.163484 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:03:16.163412    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs podName:831a5ce5-5868-4474-975e-0d678897c578 nodeName:}" failed. No retries permitted until 2026-09-11 04:03:16.663394924 +0000 UTC m=+570.927873273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs") pod "maas-controller-5d9bd4c4c-k8nr2" (UID: "831a5ce5-5868-4474-975e-0d678897c578") : secret "maas-controller-metrics-tls" not found
Sep 11 04:03:16.175221 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.175190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2ss\" (UniqueName: \"kubernetes.io/projected/831a5ce5-5868-4474-975e-0d678897c578-kube-api-access-6g2ss\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.668312 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.668279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.668496 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.668335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.670737 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.670710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.670852 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.670781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs\") pod \"maas-controller-5d9bd4c4c-k8nr2\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") " pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:16.931273 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:16.931183    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:17.202176 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:17.202151    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9bd4c4c-k8nr2"]
Sep 11 04:03:17.204664 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:03:17.204635    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831a5ce5_5868_4474_975e_0d678897c578.slice/crio-1a4315ae8cbe529979d9a4d705f348504e6dc4a214106d5b94303a4b07cfe1e5 WatchSource:0}: Error finding container 1a4315ae8cbe529979d9a4d705f348504e6dc4a214106d5b94303a4b07cfe1e5: Status 404 returned error can't find the container with id 1a4315ae8cbe529979d9a4d705f348504e6dc4a214106d5b94303a4b07cfe1e5
Sep 11 04:03:17.971754 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:17.971711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2" event={"ID":"831a5ce5-5868-4474-975e-0d678897c578","Type":"ContainerStarted","Data":"1a4315ae8cbe529979d9a4d705f348504e6dc4a214106d5b94303a4b07cfe1e5"}
Sep 11 04:03:19.982777 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:19.982730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2" event={"ID":"831a5ce5-5868-4474-975e-0d678897c578","Type":"ContainerStarted","Data":"eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989"}
Sep 11 04:03:19.983154 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:19.982807    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:20.004330 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:20.004279    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2" podStartSLOduration=2.727808923 podStartE2EDuration="5.004263519s" totalImagesPullingTime="2.276454596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:20.002783194 +0000 UTC m=+574.267261563" watchObservedRunningTime="2026-09-11 04:03:20.004263519 +0000 UTC m=+574.268741890"
Sep 11 04:03:30.609810 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:30.609773    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d9bd4c4c-k8nr2"]
Sep 11 04:03:30.610217 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:30.610016    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2" podUID="831a5ce5-5868-4474-975e-0d678897c578" containerName="manager" containerID="cri-o://eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989" gracePeriod=30
Sep 11 04:03:30.611627 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:30.611604    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:30.853023 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:30.852989    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:31.004856 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.004817    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g2ss\" (UniqueName: \"kubernetes.io/projected/831a5ce5-5868-4474-975e-0d678897c578-kube-api-access-6g2ss\") pod \"831a5ce5-5868-4474-975e-0d678897c578\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") "
Sep 11 04:03:31.005055 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.004868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs\") pod \"831a5ce5-5868-4474-975e-0d678897c578\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") "
Sep 11 04:03:31.005055 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.004906    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert\") pod \"831a5ce5-5868-4474-975e-0d678897c578\" (UID: \"831a5ce5-5868-4474-975e-0d678897c578\") "
Sep 11 04:03:31.007152 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.007115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "831a5ce5-5868-4474-975e-0d678897c578" (UID: "831a5ce5-5868-4474-975e-0d678897c578"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:31.007278 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.007164    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831a5ce5-5868-4474-975e-0d678897c578-kube-api-access-6g2ss" (OuterVolumeSpecName: "kube-api-access-6g2ss") pod "831a5ce5-5868-4474-975e-0d678897c578" (UID: "831a5ce5-5868-4474-975e-0d678897c578"). InnerVolumeSpecName "kube-api-access-6g2ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:31.007278 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.007184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "831a5ce5-5868-4474-975e-0d678897c578" (UID: "831a5ce5-5868-4474-975e-0d678897c578"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:03:31.036543 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.036510    2582 generic.go:356] "Generic (PLEG): container finished" podID="831a5ce5-5868-4474-975e-0d678897c578" containerID="eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989" exitCode=0
Sep 11 04:03:31.036543 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.036570    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2"
Sep 11 04:03:31.036819 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.036588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2" event={"ID":"831a5ce5-5868-4474-975e-0d678897c578","Type":"ContainerDied","Data":"eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989"}
Sep 11 04:03:31.036819 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.036652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9bd4c4c-k8nr2" event={"ID":"831a5ce5-5868-4474-975e-0d678897c578","Type":"ContainerDied","Data":"1a4315ae8cbe529979d9a4d705f348504e6dc4a214106d5b94303a4b07cfe1e5"}
Sep 11 04:03:31.036819 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.036683    2582 scope.go:117] "RemoveContainer" containerID="eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989"
Sep 11 04:03:31.046019 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.046000    2582 scope.go:117] "RemoveContainer" containerID="eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989"
Sep 11 04:03:31.046264 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:03:31.046247    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989\": container with ID starting with eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989 not found: ID does not exist" containerID="eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989"
Sep 11 04:03:31.046315 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.046272    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989"} err="failed to get container status \"eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989\": rpc error: code = NotFound desc = could not find container \"eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989\": container with ID starting with eccedba4c93c50c8ae510230a46c819b098505bcb4b240a070437a1b683be989 not found: ID does not exist"
Sep 11 04:03:31.057274 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.057241    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d9bd4c4c-k8nr2"]
Sep 11 04:03:31.063237 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.063216    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d9bd4c4c-k8nr2"]
Sep 11 04:03:31.106120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.106090    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-metrics-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:03:31.106120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.106117    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/831a5ce5-5868-4474-975e-0d678897c578-webhook-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:03:31.106281 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:31.106129    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g2ss\" (UniqueName: \"kubernetes.io/projected/831a5ce5-5868-4474-975e-0d678897c578-kube-api-access-6g2ss\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:03:32.274263 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:32.274233    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="831a5ce5-5868-4474-975e-0d678897c578" path="/var/lib/kubelet/pods/831a5ce5-5868-4474-975e-0d678897c578/volumes"
Sep 11 04:03:46.205605 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:46.205461    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:03:46.213570 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:46.213530    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:03:53.161874 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.161838    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"]
Sep 11 04:03:53.162327 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.162214    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="831a5ce5-5868-4474-975e-0d678897c578" containerName="manager"
Sep 11 04:03:53.162327 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.162225    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="831a5ce5-5868-4474-975e-0d678897c578" containerName="manager"
Sep 11 04:03:53.162327 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.162297    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="831a5ce5-5868-4474-975e-0d678897c578" containerName="manager"
Sep 11 04:03:53.167206 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.167173    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.170222 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.170197    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 04:03:53.171396 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.171377    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cz8bc\""
Sep 11 04:03:53.176588 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.176537    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"]
Sep 11 04:03:53.219018 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.218979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f2440-749d-46d8-bf44-05cd7e933848-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-52h68\" (UID: \"a79f2440-749d-46d8-bf44-05cd7e933848\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.219211 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.219030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhm9\" (UniqueName: \"kubernetes.io/projected/a79f2440-749d-46d8-bf44-05cd7e933848-kube-api-access-tfhm9\") pod \"payload-pre-processing-59fbc47bf8-52h68\" (UID: \"a79f2440-749d-46d8-bf44-05cd7e933848\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.319869 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.319823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f2440-749d-46d8-bf44-05cd7e933848-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-52h68\" (UID: \"a79f2440-749d-46d8-bf44-05cd7e933848\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.320065 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.319885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhm9\" (UniqueName: \"kubernetes.io/projected/a79f2440-749d-46d8-bf44-05cd7e933848-kube-api-access-tfhm9\") pod \"payload-pre-processing-59fbc47bf8-52h68\" (UID: \"a79f2440-749d-46d8-bf44-05cd7e933848\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.320660 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.320638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f2440-749d-46d8-bf44-05cd7e933848-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-52h68\" (UID: \"a79f2440-749d-46d8-bf44-05cd7e933848\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.329661 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.329621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhm9\" (UniqueName: \"kubernetes.io/projected/a79f2440-749d-46d8-bf44-05cd7e933848-kube-api-access-tfhm9\") pod \"payload-pre-processing-59fbc47bf8-52h68\" (UID: \"a79f2440-749d-46d8-bf44-05cd7e933848\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.480135 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.480092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:53.756119 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:53.756034    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"]
Sep 11 04:03:53.759716 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:03:53.759685    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79f2440_749d_46d8_bf44_05cd7e933848.slice/crio-e33c5c37a971220e733973093189029d1f219ab2aa85c65c0a0b3070db3d57d4 WatchSource:0}: Error finding container e33c5c37a971220e733973093189029d1f219ab2aa85c65c0a0b3070db3d57d4: Status 404 returned error can't find the container with id e33c5c37a971220e733973093189029d1f219ab2aa85c65c0a0b3070db3d57d4
Sep 11 04:03:54.148946 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:54.148854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68" event={"ID":"a79f2440-749d-46d8-bf44-05cd7e933848","Type":"ContainerStarted","Data":"e33c5c37a971220e733973093189029d1f219ab2aa85c65c0a0b3070db3d57d4"}
Sep 11 04:03:56.161079 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:56.161041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68" event={"ID":"a79f2440-749d-46d8-bf44-05cd7e933848","Type":"ContainerStarted","Data":"e1f6ecffe7c2a66a71c796c4cc53b5d20c73b315e1eaa0c6c4b4c862f0f10494"}
Sep 11 04:03:56.161476 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:56.161209    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:03:56.179530 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:03:56.179477    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68" podStartSLOduration=1.59986394 podStartE2EDuration="3.179459218s" totalImagesPullingTime="1.579595278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:56.178617461 +0000 UTC m=+610.443095834" watchObservedRunningTime="2026-09-11 04:03:56.179459218 +0000 UTC m=+610.443937589"
Sep 11 04:04:07.168526 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:07.168491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-52h68"
Sep 11 04:04:34.717317 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.717284    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"]
Sep 11 04:04:34.720183 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.720167    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.725426 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.725404    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 04:04:34.726482 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.726462    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-97jpl\""
Sep 11 04:04:34.726627 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.726500    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 04:04:34.726761 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.726746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 04:04:34.735610 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.735586    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"]
Sep 11 04:04:34.809819 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.809774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.809819 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.809821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.810031 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.809933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.810031 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.810006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.810102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.810041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzgj\" (UniqueName: \"kubernetes.io/projected/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-kube-api-access-jjzgj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.810102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.810075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.911145 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.911103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.911145 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.911143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.911368 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.911201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.911368 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.911245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.911469 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.911378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzgj\" (UniqueName: \"kubernetes.io/projected/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-kube-api-access-jjzgj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.911469 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.911430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.915574 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.912111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.915574 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.912232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.915574 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.912309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.915574 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.914525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.915574 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.914947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:34.920989 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:34.920961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzgj\" (UniqueName: \"kubernetes.io/projected/f9bd2e3a-d5b1-483b-9291-c2ce5481ccca-kube-api-access-jjzgj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ggzcj\" (UID: \"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:35.030955 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:35.030847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:35.305659 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:35.305630    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"]
Sep 11 04:04:35.308165 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:04:35.308138    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bd2e3a_d5b1_483b_9291_c2ce5481ccca.slice/crio-75b73ee246b7955b7939079e8209c3e8bbf704af9db40a0a838ce0f5c53bbab8 WatchSource:0}: Error finding container 75b73ee246b7955b7939079e8209c3e8bbf704af9db40a0a838ce0f5c53bbab8: Status 404 returned error can't find the container with id 75b73ee246b7955b7939079e8209c3e8bbf704af9db40a0a838ce0f5c53bbab8
Sep 11 04:04:35.309396 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:35.309378    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:04:35.352473 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:35.352436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj" event={"ID":"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca","Type":"ContainerStarted","Data":"75b73ee246b7955b7939079e8209c3e8bbf704af9db40a0a838ce0f5c53bbab8"}
Sep 11 04:04:37.364138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:37.364099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj" event={"ID":"f9bd2e3a-d5b1-483b-9291-c2ce5481ccca","Type":"ContainerStarted","Data":"1a70a9dc1ad5144c6667762388b32b533dbea78b2c3f43a0efb579c10f4a5b72"}
Sep 11 04:04:37.381450 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:37.381388    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj" podStartSLOduration=2.338536092 podStartE2EDuration="3.381366052s" totalImagesPullingTime="1.04282996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:37.380187808 +0000 UTC m=+651.644666212" watchObservedRunningTime="2026-09-11 04:04:37.381366052 +0000 UTC m=+651.645844426"
Sep 11 04:04:45.031586 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:45.031519    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:45.032124 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:45.031607    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:45.044240 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:45.044216    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:45.415829 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:45.415746    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ggzcj"
Sep 11 04:04:58.604392 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.604357    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"]
Sep 11 04:04:58.609988 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.609963    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.612412 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.612391    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 04:04:58.616929 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.616902    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"]
Sep 11 04:04:58.739481 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.739439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.739481 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.739479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.739730 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.739515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.739730 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.739622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72cd726d-f835-4d5d-be78-c604e40e3d5b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.739730 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.739654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.739730 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.739676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlx9\" (UniqueName: \"kubernetes.io/projected/72cd726d-f835-4d5d-be78-c604e40e3d5b-kube-api-access-wjlx9\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.840762 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.840708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.840762 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.840764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72cd726d-f835-4d5d-be78-c604e40e3d5b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.841004 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.840859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.841004 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.840881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlx9\" (UniqueName: \"kubernetes.io/projected/72cd726d-f835-4d5d-be78-c604e40e3d5b-kube-api-access-wjlx9\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.841120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.841025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.841120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.841048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.841234 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.841116    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.841348 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.841332    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.841387 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.841354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.843208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.843184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72cd726d-f835-4d5d-be78-c604e40e3d5b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.843335 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.843315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72cd726d-f835-4d5d-be78-c604e40e3d5b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.848412 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.848384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlx9\" (UniqueName: \"kubernetes.io/projected/72cd726d-f835-4d5d-be78-c604e40e3d5b-kube-api-access-wjlx9\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7\" (UID: \"72cd726d-f835-4d5d-be78-c604e40e3d5b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:58.922491 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:58.922455    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:04:59.195613 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:59.195517    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"]
Sep 11 04:04:59.199264 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:04:59.199228    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cd726d_f835_4d5d_be78_c604e40e3d5b.slice/crio-6eccf7c861e29d7847da39cb822d246fcf34b760831719604e784b68ac37b697 WatchSource:0}: Error finding container 6eccf7c861e29d7847da39cb822d246fcf34b760831719604e784b68ac37b697: Status 404 returned error can't find the container with id 6eccf7c861e29d7847da39cb822d246fcf34b760831719604e784b68ac37b697
Sep 11 04:04:59.480251 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:59.480160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7" event={"ID":"72cd726d-f835-4d5d-be78-c604e40e3d5b","Type":"ContainerStarted","Data":"c13868da38e87b6c047dc3a01298bb4f3a71c0cccdfc0627b1dccd1188ee9aa9"}
Sep 11 04:04:59.480251 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:59.480245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7" event={"ID":"72cd726d-f835-4d5d-be78-c604e40e3d5b","Type":"ContainerStarted","Data":"6eccf7c861e29d7847da39cb822d246fcf34b760831719604e784b68ac37b697"}
Sep 11 04:04:59.497373 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:04:59.497314    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7" podStartSLOduration=1.31127532 podStartE2EDuration="1.497297464s" totalImagesPullingTime="186.022144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:59.496602711 +0000 UTC m=+673.761081095" watchObservedRunningTime="2026-09-11 04:04:59.497297464 +0000 UTC m=+673.761775835"
Sep 11 04:05:08.922844 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:08.922796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:05:08.923237 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:08.922871    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:05:08.935198 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:08.935166    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:05:09.541544 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:09.541509    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7"
Sep 11 04:05:17.377156 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.377126    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"]
Sep 11 04:05:17.380641 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.380616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.382859 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.382837    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 04:05:17.390282 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.390249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"]
Sep 11 04:05:17.405685 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.405644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.405979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.405953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.406099 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.405992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7wx\" (UniqueName: \"kubernetes.io/projected/febe73cb-b75b-48ca-83c3-39eaabc5d62c-kube-api-access-9q7wx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.406166 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.406097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.406166 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.406134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.406262 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.406162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febe73cb-b75b-48ca-83c3-39eaabc5d62c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.507568 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.507527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.507750 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.507613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.507750 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.507645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7wx\" (UniqueName: \"kubernetes.io/projected/febe73cb-b75b-48ca-83c3-39eaabc5d62c-kube-api-access-9q7wx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.507750 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.507745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.507917 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.507781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.507917 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.507810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febe73cb-b75b-48ca-83c3-39eaabc5d62c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.508120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.508093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.508173 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.508115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.508173 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.508151    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.509895 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.509872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febe73cb-b75b-48ca-83c3-39eaabc5d62c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.510227 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.510207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febe73cb-b75b-48ca-83c3-39eaabc5d62c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.515332 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.515315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7wx\" (UniqueName: \"kubernetes.io/projected/febe73cb-b75b-48ca-83c3-39eaabc5d62c-kube-api-access-9q7wx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7\" (UID: \"febe73cb-b75b-48ca-83c3-39eaabc5d62c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.694172 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.694126    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:17.966832 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:17.966805    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"]
Sep 11 04:05:17.969652 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:05:17.969613    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebe73cb_b75b_48ca_83c3_39eaabc5d62c.slice/crio-0adc4753f5120c989d681f68ae49a2f2b57652d20b6e11c8f61b10a9389c586b WatchSource:0}: Error finding container 0adc4753f5120c989d681f68ae49a2f2b57652d20b6e11c8f61b10a9389c586b: Status 404 returned error can't find the container with id 0adc4753f5120c989d681f68ae49a2f2b57652d20b6e11c8f61b10a9389c586b
Sep 11 04:05:18.575432 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:18.575394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7" event={"ID":"febe73cb-b75b-48ca-83c3-39eaabc5d62c","Type":"ContainerStarted","Data":"e2da15ff042eff00c343189e0c8ed6cf38e4827df4db68de30d9e1ba21cfdf6c"}
Sep 11 04:05:18.575847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:18.575458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7" event={"ID":"febe73cb-b75b-48ca-83c3-39eaabc5d62c","Type":"ContainerStarted","Data":"0adc4753f5120c989d681f68ae49a2f2b57652d20b6e11c8f61b10a9389c586b"}
Sep 11 04:05:18.593219 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:18.593159    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7" podStartSLOduration=1.406077155 podStartE2EDuration="1.593140346s" totalImagesPullingTime="187.063191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:05:18.592482462 +0000 UTC m=+692.856960835" watchObservedRunningTime="2026-09-11 04:05:18.593140346 +0000 UTC m=+692.857618771"
Sep 11 04:05:27.694782 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:27.694745    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:27.695293 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:27.694827    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:27.707568 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:27.707533    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:28.635077 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:28.635047    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7"
Sep 11 04:05:38.905140 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:38.905065    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"]
Sep 11 04:05:38.908880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:38.908862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:38.911427 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:38.911404    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 04:05:38.916644 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:38.916621    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"]
Sep 11 04:05:39.024103 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.024062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375d0f4-e498-4241-98af-ac2749c1b056-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.024314 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.024114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjtpk\" (UniqueName: \"kubernetes.io/projected/7375d0f4-e498-4241-98af-ac2749c1b056-kube-api-access-rjtpk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.024314 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.024212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.024314 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.024294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.024484 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.024332    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.024484 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.024396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.125652 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.125611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.125860 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.125680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.125860 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.125715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375d0f4-e498-4241-98af-ac2749c1b056-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.125860 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.125746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjtpk\" (UniqueName: \"kubernetes.io/projected/7375d0f4-e498-4241-98af-ac2749c1b056-kube-api-access-rjtpk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.125860 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.125780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.125860 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.125824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.126124 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.126090    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.126185 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.126140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.126185 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.126141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.127984 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.127957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7375d0f4-e498-4241-98af-ac2749c1b056-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.128243 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.128225    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7375d0f4-e498-4241-98af-ac2749c1b056-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.134481 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.134446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjtpk\" (UniqueName: \"kubernetes.io/projected/7375d0f4-e498-4241-98af-ac2749c1b056-kube-api-access-rjtpk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb\" (UID: \"7375d0f4-e498-4241-98af-ac2749c1b056\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.219085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.219046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:39.492059 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.492028    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"]
Sep 11 04:05:39.494224 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:05:39.494195    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7375d0f4_e498_4241_98af_ac2749c1b056.slice/crio-f1d8443fbab54d83f739475b31be8bdfa8b6fad51bf8a8d973b75f25784d543a WatchSource:0}: Error finding container f1d8443fbab54d83f739475b31be8bdfa8b6fad51bf8a8d973b75f25784d543a: Status 404 returned error can't find the container with id f1d8443fbab54d83f739475b31be8bdfa8b6fad51bf8a8d973b75f25784d543a
Sep 11 04:05:39.682624 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:39.682586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb" event={"ID":"7375d0f4-e498-4241-98af-ac2749c1b056","Type":"ContainerStarted","Data":"f1d8443fbab54d83f739475b31be8bdfa8b6fad51bf8a8d973b75f25784d543a"}
Sep 11 04:05:40.689098 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:40.689062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb" event={"ID":"7375d0f4-e498-4241-98af-ac2749c1b056","Type":"ContainerStarted","Data":"971f7ec2edcad0e58bec1caddf2ff0a80319215a91de4dc7b3b8e48952d2d012"}
Sep 11 04:05:40.705273 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:40.705208    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb" podStartSLOduration=2.515975605 podStartE2EDuration="2.705193012s" totalImagesPullingTime="189.217407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:05:40.704895434 +0000 UTC m=+714.969373806" watchObservedRunningTime="2026-09-11 04:05:40.705193012 +0000 UTC m=+714.969671382"
Sep 11 04:05:49.219771 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:49.219728    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:49.219771 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:49.219774    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:49.232463 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:49.232433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:05:49.746021 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:05:49.745990    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb"
Sep 11 04:07:01.280644 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.280602    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58c69d57bf-cvdpl"]
Sep 11 04:07:01.281146 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.280832    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58c69d57bf-cvdpl" podUID="5cf3ac5e-4889-4363-832c-1c5ef5e8de92" containerName="authorino" containerID="cri-o://86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a" gracePeriod=30
Sep 11 04:07:01.541607 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.541533    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:07:01.543987 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.543967    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h58pd\" (UniqueName: \"kubernetes.io/projected/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-kube-api-access-h58pd\") pod \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") "
Sep 11 04:07:01.544112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.544000    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-tls-cert\") pod \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\" (UID: \"5cf3ac5e-4889-4363-832c-1c5ef5e8de92\") "
Sep 11 04:07:01.546084 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.546057    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-kube-api-access-h58pd" (OuterVolumeSpecName: "kube-api-access-h58pd") pod "5cf3ac5e-4889-4363-832c-1c5ef5e8de92" (UID: "5cf3ac5e-4889-4363-832c-1c5ef5e8de92"). InnerVolumeSpecName "kube-api-access-h58pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:01.554122 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.554099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5cf3ac5e-4889-4363-832c-1c5ef5e8de92" (UID: "5cf3ac5e-4889-4363-832c-1c5ef5e8de92"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:07:01.645288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.645247    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h58pd\" (UniqueName: \"kubernetes.io/projected/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-kube-api-access-h58pd\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:07:01.645288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:01.645294    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf3ac5e-4889-4363-832c-1c5ef5e8de92-tls-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.094127 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.094089    2582 generic.go:356] "Generic (PLEG): container finished" podID="5cf3ac5e-4889-4363-832c-1c5ef5e8de92" containerID="86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a" exitCode=0
Sep 11 04:07:02.094318 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.094140    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c69d57bf-cvdpl"
Sep 11 04:07:02.094318 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.094130    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c69d57bf-cvdpl" event={"ID":"5cf3ac5e-4889-4363-832c-1c5ef5e8de92","Type":"ContainerDied","Data":"86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a"}
Sep 11 04:07:02.094318 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.094248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c69d57bf-cvdpl" event={"ID":"5cf3ac5e-4889-4363-832c-1c5ef5e8de92","Type":"ContainerDied","Data":"d68539ef16bec560132ac6ea29d17ad1f2a8fce9c47afb3fe74fffea555f3113"}
Sep 11 04:07:02.094318 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.094273    2582 scope.go:117] "RemoveContainer" containerID="86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a"
Sep 11 04:07:02.103413 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.103397    2582 scope.go:117] "RemoveContainer" containerID="86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a"
Sep 11 04:07:02.103701 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:07:02.103683    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a\": container with ID starting with 86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a not found: ID does not exist" containerID="86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a"
Sep 11 04:07:02.103750 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.103713    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a"} err="failed to get container status \"86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a\": rpc error: code = NotFound desc = could not find container \"86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a\": container with ID starting with 86f2cc8d1199e8afef5bac057fc191122d50192fec190a4f04f4692b18e28f4a not found: ID does not exist"
Sep 11 04:07:02.116504 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.116477    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58c69d57bf-cvdpl"]
Sep 11 04:07:02.124816 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.124793    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58c69d57bf-cvdpl"]
Sep 11 04:07:02.274138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:02.274104    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf3ac5e-4889-4363-832c-1c5ef5e8de92" path="/var/lib/kubelet/pods/5cf3ac5e-4889-4363-832c-1c5ef5e8de92/volumes"
Sep 11 04:07:25.357575 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.357521    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"]
Sep 11 04:07:25.357971 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.357957    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf3ac5e-4889-4363-832c-1c5ef5e8de92" containerName="authorino"
Sep 11 04:07:25.357971 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.357972    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf3ac5e-4889-4363-832c-1c5ef5e8de92" containerName="authorino"
Sep 11 04:07:25.358062 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.358053    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf3ac5e-4889-4363-832c-1c5ef5e8de92" containerName="authorino"
Sep 11 04:07:25.361057 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.361038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.365265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.365242    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a28c9f-dockercfg-gjv57\""
Sep 11 04:07:25.365392 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.365280    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a28c9f\""
Sep 11 04:07:25.371388 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.371364    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"]
Sep 11 04:07:25.482084 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.482046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgcbt\" (UniqueName: \"kubernetes.io/projected/d86388a0-e58f-459d-a785-1e414f79083c-kube-api-access-pgcbt\") pod \"payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.482249 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.482102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86388a0-e58f-459d-a785-1e414f79083c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.583753 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.583694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgcbt\" (UniqueName: \"kubernetes.io/projected/d86388a0-e58f-459d-a785-1e414f79083c-kube-api-access-pgcbt\") pod \"payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.583753 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.583761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86388a0-e58f-459d-a785-1e414f79083c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.584438 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.584417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86388a0-e58f-459d-a785-1e414f79083c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.591389 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.591352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgcbt\" (UniqueName: \"kubernetes.io/projected/d86388a0-e58f-459d-a785-1e414f79083c-kube-api-access-pgcbt\") pod \"payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.673074 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.673043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:25.949186 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:25.949100    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"]
Sep 11 04:07:25.950116 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:07:25.950088    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86388a0_e58f_459d_a785_1e414f79083c.slice/crio-1891d0f3eaa265e9b0f8284479d5397a3435b2658c4b7289650ef10c18ce8044 WatchSource:0}: Error finding container 1891d0f3eaa265e9b0f8284479d5397a3435b2658c4b7289650ef10c18ce8044: Status 404 returned error can't find the container with id 1891d0f3eaa265e9b0f8284479d5397a3435b2658c4b7289650ef10c18ce8044
Sep 11 04:07:26.219698 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:26.219593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287" event={"ID":"d86388a0-e58f-459d-a785-1e414f79083c","Type":"ContainerStarted","Data":"a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34"}
Sep 11 04:07:26.219698 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:26.219654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287" event={"ID":"d86388a0-e58f-459d-a785-1e414f79083c","Type":"ContainerStarted","Data":"1891d0f3eaa265e9b0f8284479d5397a3435b2658c4b7289650ef10c18ce8044"}
Sep 11 04:07:26.219964 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:26.219791    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:26.237971 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:26.237913    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287" podStartSLOduration=1.237898113 podStartE2EDuration="1.237898113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:26.236136592 +0000 UTC m=+820.500614963" watchObservedRunningTime="2026-09-11 04:07:26.237898113 +0000 UTC m=+820.502376483"
Sep 11 04:07:30.795120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.795077    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"]
Sep 11 04:07:30.799228 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.799203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:30.801767 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.801734    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-dc984df1-kserve-self-signed-certs\""
Sep 11 04:07:30.808157 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.808134    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"]
Sep 11 04:07:30.941843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.941799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1464a394-b244-424f-92f4-5bc4def6a5c4-tls-certs\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:30.942057 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.941869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-model-cache\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:30.942057 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.941991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-home\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:30.942057 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.942044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49bq\" (UniqueName: \"kubernetes.io/projected/1464a394-b244-424f-92f4-5bc4def6a5c4-kube-api-access-t49bq\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:30.942201 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.942073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-tmp-dir\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:30.942201 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:30.942111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-dshm\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.042998 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.042950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1464a394-b244-424f-92f4-5bc4def6a5c4-tls-certs\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.042998 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.042999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-model-cache\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.043327 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.043036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-home\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.043327 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.043061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49bq\" (UniqueName: \"kubernetes.io/projected/1464a394-b244-424f-92f4-5bc4def6a5c4-kube-api-access-t49bq\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.043327 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.043078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-tmp-dir\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.043327 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.043102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-dshm\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.043545 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.043517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-home\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.043642 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.043621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-tmp-dir\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.043677 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.043642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-model-cache\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.045359 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.045308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-dshm\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.045626 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.045599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1464a394-b244-424f-92f4-5bc4def6a5c4-tls-certs\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.061237 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.061204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49bq\" (UniqueName: \"kubernetes.io/projected/1464a394-b244-424f-92f4-5bc4def6a5c4-kube-api-access-t49bq\") pod \"e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") " pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.110587 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.110508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:31.389350 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:31.389326    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"]
Sep 11 04:07:31.391511 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:07:31.391480    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1464a394_b244_424f_92f4_5bc4def6a5c4.slice/crio-e7dd87ef8b5f917b64afdd285af6149d94a4cb7c253002869135be2f7c7a00f4 WatchSource:0}: Error finding container e7dd87ef8b5f917b64afdd285af6149d94a4cb7c253002869135be2f7c7a00f4: Status 404 returned error can't find the container with id e7dd87ef8b5f917b64afdd285af6149d94a4cb7c253002869135be2f7c7a00f4
Sep 11 04:07:32.253102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.253065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2" event={"ID":"1464a394-b244-424f-92f4-5bc4def6a5c4","Type":"ContainerStarted","Data":"b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8"}
Sep 11 04:07:32.253102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.253107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2" event={"ID":"1464a394-b244-424f-92f4-5bc4def6a5c4","Type":"ContainerStarted","Data":"e7dd87ef8b5f917b64afdd285af6149d94a4cb7c253002869135be2f7c7a00f4"}
Sep 11 04:07:32.275501 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.275448    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2" podStartSLOduration=2.275432769 podStartE2EDuration="2.275432769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:32.273662002 +0000 UTC m=+826.538140377" watchObservedRunningTime="2026-09-11 04:07:32.275432769 +0000 UTC m=+826.539911137"
Sep 11 04:07:32.458624 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.458589    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"]
Sep 11 04:07:32.461920 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.461903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:32.464481 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.464453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-eda9e4-dockercfg-mh9kh\""
Sep 11 04:07:32.464645 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.464624    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-eda9e4\""
Sep 11 04:07:32.472233 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.472201    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"]
Sep 11 04:07:32.558703 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.558623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tb2q\" (UniqueName: \"kubernetes.io/projected/71e84f2c-9997-4107-b490-c1f711b9911d-kube-api-access-5tb2q\") pod \"payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") " pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:32.558847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.558788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e84f2c-9997-4107-b490-c1f711b9911d-plugins-config-volume\") pod \"payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") " pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:32.660111 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.660070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e84f2c-9997-4107-b490-c1f711b9911d-plugins-config-volume\") pod \"payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") " pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:32.660302 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.660127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tb2q\" (UniqueName: \"kubernetes.io/projected/71e84f2c-9997-4107-b490-c1f711b9911d-kube-api-access-5tb2q\") pod \"payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") " pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:32.660830 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.660809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e84f2c-9997-4107-b490-c1f711b9911d-plugins-config-volume\") pod \"payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") " pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:32.668297 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.668264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tb2q\" (UniqueName: \"kubernetes.io/projected/71e84f2c-9997-4107-b490-c1f711b9911d-kube-api-access-5tb2q\") pod \"payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") " pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:32.774322 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:32.774278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:33.053105 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.053081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"]
Sep 11 04:07:33.055606 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:07:33.055578    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e84f2c_9997_4107_b490_c1f711b9911d.slice/crio-8d1aee6f24636830c2f86fc352add0be0a50be297e9e4a7f4097986b31ad37e8 WatchSource:0}: Error finding container 8d1aee6f24636830c2f86fc352add0be0a50be297e9e4a7f4097986b31ad37e8: Status 404 returned error can't find the container with id 8d1aee6f24636830c2f86fc352add0be0a50be297e9e4a7f4097986b31ad37e8
Sep 11 04:07:33.260943 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.260907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5" event={"ID":"71e84f2c-9997-4107-b490-c1f711b9911d","Type":"ContainerStarted","Data":"73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a"}
Sep 11 04:07:33.261422 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.260964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5" event={"ID":"71e84f2c-9997-4107-b490-c1f711b9911d","Type":"ContainerStarted","Data":"8d1aee6f24636830c2f86fc352add0be0a50be297e9e4a7f4097986b31ad37e8"}
Sep 11 04:07:33.261422 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.261193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:33.279436 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.279380    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5" podStartSLOduration=1.279365829 podStartE2EDuration="1.279365829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:33.277066464 +0000 UTC m=+827.541544833" watchObservedRunningTime="2026-09-11 04:07:33.279365829 +0000 UTC m=+827.543844199"
Sep 11 04:07:33.453769 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.453737    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"]
Sep 11 04:07:33.457932 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.457905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:33.462186 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.462146    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d6904f79\""
Sep 11 04:07:33.462318 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.462258    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d6904f79-dockercfg-5hdsf\""
Sep 11 04:07:33.469607 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.469581    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"]
Sep 11 04:07:33.567291 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.567253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04b9aff-1a45-4027-bcf6-05e5ea379587-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:33.567471 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.567314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grd8\" (UniqueName: \"kubernetes.io/projected/a04b9aff-1a45-4027-bcf6-05e5ea379587-kube-api-access-6grd8\") pod \"payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:33.668612 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.668531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grd8\" (UniqueName: \"kubernetes.io/projected/a04b9aff-1a45-4027-bcf6-05e5ea379587-kube-api-access-6grd8\") pod \"payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:33.668783 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.668711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04b9aff-1a45-4027-bcf6-05e5ea379587-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:33.676647 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.669837    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04b9aff-1a45-4027-bcf6-05e5ea379587-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:33.679641 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.679613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grd8\" (UniqueName: \"kubernetes.io/projected/a04b9aff-1a45-4027-bcf6-05e5ea379587-kube-api-access-6grd8\") pod \"payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:33.769693 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:33.769588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:34.043962 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:34.043936    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"]
Sep 11 04:07:34.045438 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:07:34.045410    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04b9aff_1a45_4027_bcf6_05e5ea379587.slice/crio-7623f25b3005989cf08b20a23c2a115aa7dc83edee2a3699cd08ce8a98a326a8 WatchSource:0}: Error finding container 7623f25b3005989cf08b20a23c2a115aa7dc83edee2a3699cd08ce8a98a326a8: Status 404 returned error can't find the container with id 7623f25b3005989cf08b20a23c2a115aa7dc83edee2a3699cd08ce8a98a326a8
Sep 11 04:07:34.275487 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:34.275449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p" event={"ID":"a04b9aff-1a45-4027-bcf6-05e5ea379587","Type":"ContainerStarted","Data":"1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f"}
Sep 11 04:07:34.275882 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:34.275504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p" event={"ID":"a04b9aff-1a45-4027-bcf6-05e5ea379587","Type":"ContainerStarted","Data":"7623f25b3005989cf08b20a23c2a115aa7dc83edee2a3699cd08ce8a98a326a8"}
Sep 11 04:07:34.288980 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:34.288927    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p" podStartSLOduration=1.288908244 podStartE2EDuration="1.288908244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:34.288012471 +0000 UTC m=+828.552490845" watchObservedRunningTime="2026-09-11 04:07:34.288908244 +0000 UTC m=+828.553386618"
Sep 11 04:07:35.275621 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:35.275583    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:37.228240 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:37.228208    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:07:41.110848 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:41.110797    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:41.110848 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:41.110850    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:41.123660 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:41.123622    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:41.316887 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:41.316845    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:07:44.276478 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:44.276450    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:07:46.282123 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:46.282094    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:07:54.703304 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.703268    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"]
Sep 11 04:07:54.707909 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.707886    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:54.709525 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.709501    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"]
Sep 11 04:07:54.710440 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.710393    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fca08f-dockercfg-s6kbr\""
Sep 11 04:07:54.711178 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.711157    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fca08f\""
Sep 11 04:07:54.713110 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.713094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:54.719999 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.719966    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"]
Sep 11 04:07:54.726382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.726358    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"]
Sep 11 04:07:54.786340 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.786303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5nm\" (UniqueName: \"kubernetes.io/projected/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-kube-api-access-kl5nm\") pod \"payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:54.786516 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.786459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssh7p\" (UniqueName: \"kubernetes.io/projected/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-kube-api-access-ssh7p\") pod \"payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:54.786596 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.786515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:54.786596 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.786539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:54.887310 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.887267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5nm\" (UniqueName: \"kubernetes.io/projected/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-kube-api-access-kl5nm\") pod \"payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:54.887502 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.887365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssh7p\" (UniqueName: \"kubernetes.io/projected/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-kube-api-access-ssh7p\") pod \"payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:54.887502 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.887408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:54.887502 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.887454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:54.888225 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.888200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:54.888324 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.888199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:54.909790 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.909758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5nm\" (UniqueName: \"kubernetes.io/projected/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-kube-api-access-kl5nm\") pod \"payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:54.909909 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:54.909807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssh7p\" (UniqueName: \"kubernetes.io/projected/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-kube-api-access-ssh7p\") pod \"payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:55.021377 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.021280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:55.028466 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.028442    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:55.306019 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.305994    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"]
Sep 11 04:07:55.308515 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:07:55.308489    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1bad69_e18d_4ab0_b36a_49ec7c2b6da4.slice/crio-18f234c811cf2d603e1dfe71b47052f3a08834f9f6bfe6c4bc9ffa3b270e8ff1 WatchSource:0}: Error finding container 18f234c811cf2d603e1dfe71b47052f3a08834f9f6bfe6c4bc9ffa3b270e8ff1: Status 404 returned error can't find the container with id 18f234c811cf2d603e1dfe71b47052f3a08834f9f6bfe6c4bc9ffa3b270e8ff1
Sep 11 04:07:55.320035 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.320014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"]
Sep 11 04:07:55.322857 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:07:55.322829    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b508ed_7e75_4fe4_bdb1_d37320b9fed7.slice/crio-3f9d91882347e8decea105c6ce5940e28b1218e73a8a806ebd2bffa92d99622a WatchSource:0}: Error finding container 3f9d91882347e8decea105c6ce5940e28b1218e73a8a806ebd2bffa92d99622a: Status 404 returned error can't find the container with id 3f9d91882347e8decea105c6ce5940e28b1218e73a8a806ebd2bffa92d99622a
Sep 11 04:07:55.384218 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.384050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl" event={"ID":"46b508ed-7e75-4fe4-bdb1-d37320b9fed7","Type":"ContainerStarted","Data":"7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7"}
Sep 11 04:07:55.384218 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.384151    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl" event={"ID":"46b508ed-7e75-4fe4-bdb1-d37320b9fed7","Type":"ContainerStarted","Data":"3f9d91882347e8decea105c6ce5940e28b1218e73a8a806ebd2bffa92d99622a"}
Sep 11 04:07:55.384218 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.384188    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:07:55.386079 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.386057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9" event={"ID":"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4","Type":"ContainerStarted","Data":"7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521"}
Sep 11 04:07:55.386178 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.386095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9" event={"ID":"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4","Type":"ContainerStarted","Data":"18f234c811cf2d603e1dfe71b47052f3a08834f9f6bfe6c4bc9ffa3b270e8ff1"}
Sep 11 04:07:55.386178 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.386157    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:07:55.404970 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.404901    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl" podStartSLOduration=1.404877607 podStartE2EDuration="1.404877607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:55.4017615 +0000 UTC m=+849.666239872" watchObservedRunningTime="2026-09-11 04:07:55.404877607 +0000 UTC m=+849.669355979"
Sep 11 04:07:55.421134 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:07:55.421081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9" podStartSLOduration=1.421063448 podStartE2EDuration="1.421063448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:55.420359088 +0000 UTC m=+849.684837463" watchObservedRunningTime="2026-09-11 04:07:55.421063448 +0000 UTC m=+849.685541819"
Sep 11 04:08:06.393128 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:06.393092    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:08:06.393642 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:06.393245    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:08:13.195863 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.195821    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"]
Sep 11 04:08:13.196760 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.196724    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p" podUID="a04b9aff-1a45-4027-bcf6-05e5ea379587" containerName="payload-pre-processing" containerID="cri-o://1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f" gracePeriod=30
Sep 11 04:08:13.448503 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.448445    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:08:13.484782 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.484674    2582 generic.go:356] "Generic (PLEG): container finished" podID="a04b9aff-1a45-4027-bcf6-05e5ea379587" containerID="1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f" exitCode=0
Sep 11 04:08:13.484954 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.484808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p" event={"ID":"a04b9aff-1a45-4027-bcf6-05e5ea379587","Type":"ContainerDied","Data":"1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f"}
Sep 11 04:08:13.484954 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.484901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p" event={"ID":"a04b9aff-1a45-4027-bcf6-05e5ea379587","Type":"ContainerDied","Data":"7623f25b3005989cf08b20a23c2a115aa7dc83edee2a3699cd08ce8a98a326a8"}
Sep 11 04:08:13.485080 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.484957    2582 scope.go:117] "RemoveContainer" containerID="1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f"
Sep 11 04:08:13.485080 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.484774    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"
Sep 11 04:08:13.497357 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.497334    2582 scope.go:117] "RemoveContainer" containerID="1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f"
Sep 11 04:08:13.497809 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:08:13.497775    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f\": container with ID starting with 1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f not found: ID does not exist" containerID="1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f"
Sep 11 04:08:13.497900 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.497822    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f"} err="failed to get container status \"1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f\": rpc error: code = NotFound desc = could not find container \"1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f\": container with ID starting with 1eb6f6f7ce06b032f4446c9eee2ba43b065f1edd7fd372481d39b9719db2f61f not found: ID does not exist"
Sep 11 04:08:13.577217 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.577185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6grd8\" (UniqueName: \"kubernetes.io/projected/a04b9aff-1a45-4027-bcf6-05e5ea379587-kube-api-access-6grd8\") pod \"a04b9aff-1a45-4027-bcf6-05e5ea379587\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") "
Sep 11 04:08:13.577377 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.577231    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04b9aff-1a45-4027-bcf6-05e5ea379587-plugins-config-volume\") pod \"a04b9aff-1a45-4027-bcf6-05e5ea379587\" (UID: \"a04b9aff-1a45-4027-bcf6-05e5ea379587\") "
Sep 11 04:08:13.577685 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.577663    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04b9aff-1a45-4027-bcf6-05e5ea379587-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a04b9aff-1a45-4027-bcf6-05e5ea379587" (UID: "a04b9aff-1a45-4027-bcf6-05e5ea379587"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:13.579382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.579350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04b9aff-1a45-4027-bcf6-05e5ea379587-kube-api-access-6grd8" (OuterVolumeSpecName: "kube-api-access-6grd8") pod "a04b9aff-1a45-4027-bcf6-05e5ea379587" (UID: "a04b9aff-1a45-4027-bcf6-05e5ea379587"). InnerVolumeSpecName "kube-api-access-6grd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:13.678114 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.678081    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grd8\" (UniqueName: \"kubernetes.io/projected/a04b9aff-1a45-4027-bcf6-05e5ea379587-kube-api-access-6grd8\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:08:13.678114 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.678109    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04b9aff-1a45-4027-bcf6-05e5ea379587-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:08:13.808821 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.808788    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"]
Sep 11 04:08:13.812688 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:13.812661    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d6904f79-85c7f8f7c8-sxn7p"]
Sep 11 04:08:14.274915 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:14.274882    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a04b9aff-1a45-4027-bcf6-05e5ea379587" path="/var/lib/kubelet/pods/a04b9aff-1a45-4027-bcf6-05e5ea379587/volumes"
Sep 11 04:08:18.391453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.391415    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"]
Sep 11 04:08:18.391994 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.391977    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04b9aff-1a45-4027-bcf6-05e5ea379587" containerName="payload-pre-processing"
Sep 11 04:08:18.392044 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.391996    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04b9aff-1a45-4027-bcf6-05e5ea379587" containerName="payload-pre-processing"
Sep 11 04:08:18.392093 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.392084    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04b9aff-1a45-4027-bcf6-05e5ea379587" containerName="payload-pre-processing"
Sep 11 04:08:18.396688 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.396671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.399383 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.399360    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-52e7067f\""
Sep 11 04:08:18.399548 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.399451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-52e7067f-dockercfg-6j5qm\""
Sep 11 04:08:18.409343 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.409321    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"]
Sep 11 04:08:18.525142 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.525103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8aea3b-193d-4735-b144-508a093dc893-plugins-config-volume\") pod \"payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") " pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.525142 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.525146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hsj\" (UniqueName: \"kubernetes.io/projected/3d8aea3b-193d-4735-b144-508a093dc893-kube-api-access-h4hsj\") pod \"payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") " pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.626215 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.626151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8aea3b-193d-4735-b144-508a093dc893-plugins-config-volume\") pod \"payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") " pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.626215 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.626201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hsj\" (UniqueName: \"kubernetes.io/projected/3d8aea3b-193d-4735-b144-508a093dc893-kube-api-access-h4hsj\") pod \"payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") " pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.627197 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.627170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8aea3b-193d-4735-b144-508a093dc893-plugins-config-volume\") pod \"payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") " pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.634969 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.634937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hsj\" (UniqueName: \"kubernetes.io/projected/3d8aea3b-193d-4735-b144-508a093dc893-kube-api-access-h4hsj\") pod \"payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") " pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.707906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.707875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:18.986057 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:18.986026    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"]
Sep 11 04:08:18.987933 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:08:18.987903    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8aea3b_193d_4735_b144_508a093dc893.slice/crio-c3abbf83c73ae9bf84ad1d966347a5de95735c588e8c0d79061a6f645d207249 WatchSource:0}: Error finding container c3abbf83c73ae9bf84ad1d966347a5de95735c588e8c0d79061a6f645d207249: Status 404 returned error can't find the container with id c3abbf83c73ae9bf84ad1d966347a5de95735c588e8c0d79061a6f645d207249
Sep 11 04:08:19.524347 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:19.524308    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq" event={"ID":"3d8aea3b-193d-4735-b144-508a093dc893","Type":"ContainerStarted","Data":"a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a"}
Sep 11 04:08:19.524742 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:19.524363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq" event={"ID":"3d8aea3b-193d-4735-b144-508a093dc893","Type":"ContainerStarted","Data":"c3abbf83c73ae9bf84ad1d966347a5de95735c588e8c0d79061a6f645d207249"}
Sep 11 04:08:19.524742 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:19.524405    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:19.553739 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:19.553685    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq" podStartSLOduration=1.553668587 podStartE2EDuration="1.553668587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:19.552690438 +0000 UTC m=+873.817168809" watchObservedRunningTime="2026-09-11 04:08:19.553668587 +0000 UTC m=+873.818146958"
Sep 11 04:08:30.532235 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:30.532203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:42.321099 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.321061    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"]
Sep 11 04:08:42.324727 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.324707    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"
Sep 11 04:08:42.327410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.327389    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:08:42.327683 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.327658    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:08:42.332415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.332389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"]
Sep 11 04:08:42.459068 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.459039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb2b37a-a1ae-4be8-a14e-541f844d7e9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8\" (UID: \"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"
Sep 11 04:08:42.560448 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.560404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb2b37a-a1ae-4be8-a14e-541f844d7e9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8\" (UID: \"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"
Sep 11 04:08:42.561073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.561051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb2b37a-a1ae-4be8-a14e-541f844d7e9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8\" (UID: \"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"
Sep 11 04:08:42.637174 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.637091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"
Sep 11 04:08:42.912046 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:42.912019    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"]
Sep 11 04:08:42.914232 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:08:42.914201    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb2b37a_a1ae_4be8_a14e_541f844d7e9b.slice/crio-9ce6effd8d18e82da345bdd85ea7af3826a3ada1d67ffb846bec65257094aa4a WatchSource:0}: Error finding container 9ce6effd8d18e82da345bdd85ea7af3826a3ada1d67ffb846bec65257094aa4a: Status 404 returned error can't find the container with id 9ce6effd8d18e82da345bdd85ea7af3826a3ada1d67ffb846bec65257094aa4a
Sep 11 04:08:43.656432 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:43.656386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8" event={"ID":"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b","Type":"ContainerStarted","Data":"9ce6effd8d18e82da345bdd85ea7af3826a3ada1d67ffb846bec65257094aa4a"}
Sep 11 04:08:45.670289 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:45.670247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8" event={"ID":"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b","Type":"ContainerStarted","Data":"e3a242dcffcdb8c53d7024191eb09e627bf9e1cd349b6ef5100043830dfd94c0"}
Sep 11 04:08:45.695351 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:45.695299    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8" podStartSLOduration=1.5048085310000001 podStartE2EDuration="3.695284848s" totalImagesPullingTime="2.190476317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:45.692674156 +0000 UTC m=+899.957152528" watchObservedRunningTime="2026-09-11 04:08:45.695284848 +0000 UTC m=+899.959763218"
Sep 11 04:08:46.251027 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:46.250917    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:08:46.262448 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:46.251129    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:08:46.676399 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:46.676360    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecb2b37a-a1ae-4be8-a14e-541f844d7e9b" containerID="e3a242dcffcdb8c53d7024191eb09e627bf9e1cd349b6ef5100043830dfd94c0" exitCode=0
Sep 11 04:08:46.676858 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:46.676444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8" event={"ID":"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b","Type":"ContainerDied","Data":"e3a242dcffcdb8c53d7024191eb09e627bf9e1cd349b6ef5100043830dfd94c0"}
Sep 11 04:08:47.828175 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:47.828148    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"
Sep 11 04:08:47.913918 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:47.913883    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb2b37a-a1ae-4be8-a14e-541f844d7e9b-maas-api-service-ca\") pod \"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b\" (UID: \"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b\") "
Sep 11 04:08:47.914234 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:47.914210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb2b37a-a1ae-4be8-a14e-541f844d7e9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ecb2b37a-a1ae-4be8-a14e-541f844d7e9b" (UID: "ecb2b37a-a1ae-4be8-a14e-541f844d7e9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:48.015442 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:48.015353    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb2b37a-a1ae-4be8-a14e-541f844d7e9b-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:08:48.688992 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:48.688963    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"
Sep 11 04:08:48.689175 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:48.688991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8" event={"ID":"ecb2b37a-a1ae-4be8-a14e-541f844d7e9b","Type":"ContainerDied","Data":"9ce6effd8d18e82da345bdd85ea7af3826a3ada1d67ffb846bec65257094aa4a"}
Sep 11 04:08:48.689175 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:48.689042    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ce6effd8d18e82da345bdd85ea7af3826a3ada1d67ffb846bec65257094aa4a"
Sep 11 04:08:59.346731 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.346695    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"]
Sep 11 04:08:59.347191 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.347055    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq" podUID="3d8aea3b-193d-4735-b144-508a093dc893" containerName="payload-processing" containerID="cri-o://a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a" gracePeriod=30
Sep 11 04:08:59.596932 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.596857    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:59.630916 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.630878    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8aea3b-193d-4735-b144-508a093dc893-plugins-config-volume\") pod \"3d8aea3b-193d-4735-b144-508a093dc893\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") "
Sep 11 04:08:59.631085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.630973    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4hsj\" (UniqueName: \"kubernetes.io/projected/3d8aea3b-193d-4735-b144-508a093dc893-kube-api-access-h4hsj\") pod \"3d8aea3b-193d-4735-b144-508a093dc893\" (UID: \"3d8aea3b-193d-4735-b144-508a093dc893\") "
Sep 11 04:08:59.631085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.630881    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"]
Sep 11 04:08:59.631222 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.631198    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8aea3b-193d-4735-b144-508a093dc893-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d8aea3b-193d-4735-b144-508a093dc893" (UID: "3d8aea3b-193d-4735-b144-508a093dc893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:08:59.631345 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.631330    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8aea3b-193d-4735-b144-508a093dc893-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.633231 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.633207    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8aea3b-193d-4735-b144-508a093dc893-kube-api-access-h4hsj" (OuterVolumeSpecName: "kube-api-access-h4hsj") pod "3d8aea3b-193d-4735-b144-508a093dc893" (UID: "3d8aea3b-193d-4735-b144-508a093dc893"). InnerVolumeSpecName "kube-api-access-h4hsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:59.639086 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.639047    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-52e7067f-5q7j8"]
Sep 11 04:08:59.732051 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.732004    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hsj\" (UniqueName: \"kubernetes.io/projected/3d8aea3b-193d-4735-b144-508a093dc893-kube-api-access-h4hsj\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:08:59.758995 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.758963    2582 generic.go:356] "Generic (PLEG): container finished" podID="3d8aea3b-193d-4735-b144-508a093dc893" containerID="a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a" exitCode=0
Sep 11 04:08:59.759185 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.759013    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"
Sep 11 04:08:59.759185 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.759045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq" event={"ID":"3d8aea3b-193d-4735-b144-508a093dc893","Type":"ContainerDied","Data":"a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a"}
Sep 11 04:08:59.759185 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.759116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq" event={"ID":"3d8aea3b-193d-4735-b144-508a093dc893","Type":"ContainerDied","Data":"c3abbf83c73ae9bf84ad1d966347a5de95735c588e8c0d79061a6f645d207249"}
Sep 11 04:08:59.759185 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.759147    2582 scope.go:117] "RemoveContainer" containerID="a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a"
Sep 11 04:08:59.770267 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.770242    2582 scope.go:117] "RemoveContainer" containerID="a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a"
Sep 11 04:08:59.770612 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:08:59.770590    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a\": container with ID starting with a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a not found: ID does not exist" containerID="a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a"
Sep 11 04:08:59.770690 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.770621    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a"} err="failed to get container status \"a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a\": rpc error: code = NotFound desc = could not find container \"a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a\": container with ID starting with a4c283a97e823cd5fdeffbd00bed5f84b42ab14c888a20647d05a842e7aaa18a not found: ID does not exist"
Sep 11 04:08:59.793898 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.793862    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"]
Sep 11 04:08:59.807807 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:08:59.807777    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-52e7067f-f84f59c74-9l4mq"]
Sep 11 04:09:00.275221 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:00.275191    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8aea3b-193d-4735-b144-508a093dc893" path="/var/lib/kubelet/pods/3d8aea3b-193d-4735-b144-508a093dc893/volumes"
Sep 11 04:09:00.275656 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:00.275641    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb2b37a-a1ae-4be8-a14e-541f844d7e9b" path="/var/lib/kubelet/pods/ecb2b37a-a1ae-4be8-a14e-541f844d7e9b/volumes"
Sep 11 04:09:04.867434 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.867395    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"]
Sep 11 04:09:04.867864 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.867832    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb2b37a-a1ae-4be8-a14e-541f844d7e9b" containerName="revoke-keys"
Sep 11 04:09:04.867864 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.867843    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb2b37a-a1ae-4be8-a14e-541f844d7e9b" containerName="revoke-keys"
Sep 11 04:09:04.867864 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.867863    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8aea3b-193d-4735-b144-508a093dc893" containerName="payload-processing"
Sep 11 04:09:04.867975 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.867868    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8aea3b-193d-4735-b144-508a093dc893" containerName="payload-processing"
Sep 11 04:09:04.867975 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.867955    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb2b37a-a1ae-4be8-a14e-541f844d7e9b" containerName="revoke-keys"
Sep 11 04:09:04.867975 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.867963    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8aea3b-193d-4735-b144-508a093dc893" containerName="payload-processing"
Sep 11 04:09:04.875076 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.875056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:04.877761 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.877741    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-26814939\""
Sep 11 04:09:04.877836 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.877827    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-26814939-dockercfg-gwnxw\""
Sep 11 04:09:04.887527 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.887503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"]
Sep 11 04:09:04.985720 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.985675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c744146a-912f-4174-9c2c-e1fdedae9582-plugins-config-volume\") pod \"payload-processing-e2e-ait-26814939-784f95c975-cgtc9\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") " pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:04.985865 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:04.985731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdb5m\" (UniqueName: \"kubernetes.io/projected/c744146a-912f-4174-9c2c-e1fdedae9582-kube-api-access-tdb5m\") pod \"payload-processing-e2e-ait-26814939-784f95c975-cgtc9\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") " pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:05.087038 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.086998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c744146a-912f-4174-9c2c-e1fdedae9582-plugins-config-volume\") pod \"payload-processing-e2e-ait-26814939-784f95c975-cgtc9\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") " pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:05.087216 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.087048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdb5m\" (UniqueName: \"kubernetes.io/projected/c744146a-912f-4174-9c2c-e1fdedae9582-kube-api-access-tdb5m\") pod \"payload-processing-e2e-ait-26814939-784f95c975-cgtc9\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") " pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:05.087730 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.087709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c744146a-912f-4174-9c2c-e1fdedae9582-plugins-config-volume\") pod \"payload-processing-e2e-ait-26814939-784f95c975-cgtc9\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") " pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:05.100965 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.100937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdb5m\" (UniqueName: \"kubernetes.io/projected/c744146a-912f-4174-9c2c-e1fdedae9582-kube-api-access-tdb5m\") pod \"payload-processing-e2e-ait-26814939-784f95c975-cgtc9\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") " pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:05.185757 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.185725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:05.496937 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.496907    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"]
Sep 11 04:09:05.498102 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:09:05.498074    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc744146a_912f_4174_9c2c_e1fdedae9582.slice/crio-9d9c8c53c1d9e0c1605d79ec5ffd9675666e5a30d287eb7ae48f9d00bf828132 WatchSource:0}: Error finding container 9d9c8c53c1d9e0c1605d79ec5ffd9675666e5a30d287eb7ae48f9d00bf828132: Status 404 returned error can't find the container with id 9d9c8c53c1d9e0c1605d79ec5ffd9675666e5a30d287eb7ae48f9d00bf828132
Sep 11 04:09:05.795073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.794972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9" event={"ID":"c744146a-912f-4174-9c2c-e1fdedae9582","Type":"ContainerStarted","Data":"ed7b8a47d7143ad6c654361db985e6b522a4eca054da0c8072a4754959a948a4"}
Sep 11 04:09:05.795073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.795038    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:05.795073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.795063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9" event={"ID":"c744146a-912f-4174-9c2c-e1fdedae9582","Type":"ContainerStarted","Data":"9d9c8c53c1d9e0c1605d79ec5ffd9675666e5a30d287eb7ae48f9d00bf828132"}
Sep 11 04:09:05.822967 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:05.822912    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9" podStartSLOduration=1.822895333 podStartE2EDuration="1.822895333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:09:05.820740754 +0000 UTC m=+920.085219130" watchObservedRunningTime="2026-09-11 04:09:05.822895333 +0000 UTC m=+920.087373703"
Sep 11 04:09:08.008115 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.008077    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"]
Sep 11 04:09:08.012073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.012056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.015509 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.015490    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-dc984df1-kserve-self-signed-certs\""
Sep 11 04:09:08.028018 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.027994    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"]
Sep 11 04:09:08.122059 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.122003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tmp-dir\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.122059 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.122069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-dshm\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.122282 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.122122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-model-cache\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.122282 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.122192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-home\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.122282 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.122215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tls-certs\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.122282 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.122247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jqf\" (UniqueName: \"kubernetes.io/projected/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-kube-api-access-g8jqf\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224275 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-model-cache\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-home\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tls-certs\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jqf\" (UniqueName: \"kubernetes.io/projected/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-kube-api-access-g8jqf\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tmp-dir\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224678 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-dshm\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224745 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-model-cache\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224804 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-home\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.224966 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.224939    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tmp-dir\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.226650 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.226632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-dshm\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.227034 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.227012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tls-certs\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.235380 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.235344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jqf\" (UniqueName: \"kubernetes.io/projected/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-kube-api-access-g8jqf\") pod \"e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") " pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.321990 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.321897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:08.606684 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.606652    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"]
Sep 11 04:09:08.608272 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:09:08.608246    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee1dcade_791a_4dbc_9bef_de7cdfbf6997.slice/crio-6421a466bb2d9f4da739e59185c61030bf64d1b8914562d216a57cff2086931b WatchSource:0}: Error finding container 6421a466bb2d9f4da739e59185c61030bf64d1b8914562d216a57cff2086931b: Status 404 returned error can't find the container with id 6421a466bb2d9f4da739e59185c61030bf64d1b8914562d216a57cff2086931b
Sep 11 04:09:08.813422 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.813386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf" event={"ID":"ee1dcade-791a-4dbc-9bef-de7cdfbf6997","Type":"ContainerStarted","Data":"f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482"}
Sep 11 04:09:08.813422 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.813430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf" event={"ID":"ee1dcade-791a-4dbc-9bef-de7cdfbf6997","Type":"ContainerStarted","Data":"6421a466bb2d9f4da739e59185c61030bf64d1b8914562d216a57cff2086931b"}
Sep 11 04:09:08.835226 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:08.835176    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf" podStartSLOduration=1.8351602649999998 podStartE2EDuration="1.835160265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:09:08.833937794 +0000 UTC m=+923.098416170" watchObservedRunningTime="2026-09-11 04:09:08.835160265 +0000 UTC m=+923.099638636"
Sep 11 04:09:16.802512 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:16.802474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:18.322720 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:18.322682    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:18.323281 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:18.323263    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:18.348544 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:18.348513    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:18.885241 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:18.885203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:09:32.501457 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.501418    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"]
Sep 11 04:09:32.514433 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.514396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.518132 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.518104    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-eda9e4\"/\"kube-root-ca.crt\""
Sep 11 04:09:32.518290 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.518114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-eda9e4\"/\"autores-model-eda9e4-kserve-self-signed-certs\""
Sep 11 04:09:32.518290 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.518193    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-eda9e4\"/\"openshift-service-ca.crt\""
Sep 11 04:09:32.518290 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.518126    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-eda9e4\"/\"default-dockercfg-kzstn\""
Sep 11 04:09:32.518480 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.518455    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"]
Sep 11 04:09:32.572213 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.572173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173797d7-69ae-48e5-acad-47e36d9909b6-tls-certs\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.572213 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.572214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6pn\" (UniqueName: \"kubernetes.io/projected/173797d7-69ae-48e5-acad-47e36d9909b6-kube-api-access-fj6pn\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.572425 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.572275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-home\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.572425 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.572315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-dshm\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.572425 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.572409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-tmp-dir\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.572549 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.572486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-model-cache\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.673942 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.673904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173797d7-69ae-48e5-acad-47e36d9909b6-tls-certs\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.673942 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.673945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6pn\" (UniqueName: \"kubernetes.io/projected/173797d7-69ae-48e5-acad-47e36d9909b6-kube-api-access-fj6pn\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.674209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.673974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-home\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.674209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.674000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-dshm\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.674209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.674024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-tmp-dir\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.674209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.674067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-model-cache\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.674459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.674437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-home\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.674507 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.674475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-model-cache\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.674582 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.674543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-tmp-dir\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.676310 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.676293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-dshm\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.676594 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.676574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173797d7-69ae-48e5-acad-47e36d9909b6-tls-certs\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.683127 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.683104    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6pn\" (UniqueName: \"kubernetes.io/projected/173797d7-69ae-48e5-acad-47e36d9909b6-kube-api-access-fj6pn\") pod \"autores-model-eda9e4-kserve-86bfb765bc-dkzbw\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") " pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:32.826694 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:32.826605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:33.101188 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:33.101157    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"]
Sep 11 04:09:33.103256 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:09:33.103227    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod173797d7_69ae_48e5_acad_47e36d9909b6.slice/crio-e9d529f2aefedf39bc227f0e9c6a3d2e5bdc9e707fc46bf1f223635caab7cef0 WatchSource:0}: Error finding container e9d529f2aefedf39bc227f0e9c6a3d2e5bdc9e707fc46bf1f223635caab7cef0: Status 404 returned error can't find the container with id e9d529f2aefedf39bc227f0e9c6a3d2e5bdc9e707fc46bf1f223635caab7cef0
Sep 11 04:09:33.958036 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:33.958001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw" event={"ID":"173797d7-69ae-48e5-acad-47e36d9909b6","Type":"ContainerStarted","Data":"014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059"}
Sep 11 04:09:33.958036 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:33.958042    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw" event={"ID":"173797d7-69ae-48e5-acad-47e36d9909b6","Type":"ContainerStarted","Data":"e9d529f2aefedf39bc227f0e9c6a3d2e5bdc9e707fc46bf1f223635caab7cef0"}
Sep 11 04:09:33.978717 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:33.978664    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw" podStartSLOduration=1.978648258 podStartE2EDuration="1.978648258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:09:33.978079038 +0000 UTC m=+948.242557408" watchObservedRunningTime="2026-09-11 04:09:33.978648258 +0000 UTC m=+948.243126629"
Sep 11 04:09:34.709228 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.709193    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"]
Sep 11 04:09:34.709477 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.709449    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9" podUID="c744146a-912f-4174-9c2c-e1fdedae9582" containerName="payload-processing" containerID="cri-o://ed7b8a47d7143ad6c654361db985e6b522a4eca054da0c8072a4754959a948a4" gracePeriod=30
Sep 11 04:09:34.965168 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.965093    2582 generic.go:356] "Generic (PLEG): container finished" podID="c744146a-912f-4174-9c2c-e1fdedae9582" containerID="ed7b8a47d7143ad6c654361db985e6b522a4eca054da0c8072a4754959a948a4" exitCode=0
Sep 11 04:09:34.965516 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.965167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9" event={"ID":"c744146a-912f-4174-9c2c-e1fdedae9582","Type":"ContainerDied","Data":"ed7b8a47d7143ad6c654361db985e6b522a4eca054da0c8072a4754959a948a4"}
Sep 11 04:09:34.965516 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.965235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9" event={"ID":"c744146a-912f-4174-9c2c-e1fdedae9582","Type":"ContainerDied","Data":"9d9c8c53c1d9e0c1605d79ec5ffd9675666e5a30d287eb7ae48f9d00bf828132"}
Sep 11 04:09:34.965516 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.965280    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d9c8c53c1d9e0c1605d79ec5ffd9675666e5a30d287eb7ae48f9d00bf828132"
Sep 11 04:09:34.974652 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.974630    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:34.998805 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.998776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdb5m\" (UniqueName: \"kubernetes.io/projected/c744146a-912f-4174-9c2c-e1fdedae9582-kube-api-access-tdb5m\") pod \"c744146a-912f-4174-9c2c-e1fdedae9582\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") "
Sep 11 04:09:34.998967 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.998938    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c744146a-912f-4174-9c2c-e1fdedae9582-plugins-config-volume\") pod \"c744146a-912f-4174-9c2c-e1fdedae9582\" (UID: \"c744146a-912f-4174-9c2c-e1fdedae9582\") "
Sep 11 04:09:34.999653 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:34.999407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c744146a-912f-4174-9c2c-e1fdedae9582-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c744146a-912f-4174-9c2c-e1fdedae9582" (UID: "c744146a-912f-4174-9c2c-e1fdedae9582"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:35.001595 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:35.001538    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c744146a-912f-4174-9c2c-e1fdedae9582-kube-api-access-tdb5m" (OuterVolumeSpecName: "kube-api-access-tdb5m") pod "c744146a-912f-4174-9c2c-e1fdedae9582" (UID: "c744146a-912f-4174-9c2c-e1fdedae9582"). InnerVolumeSpecName "kube-api-access-tdb5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:35.100146 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:35.100109    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c744146a-912f-4174-9c2c-e1fdedae9582-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.100146 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:35.100140    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdb5m\" (UniqueName: \"kubernetes.io/projected/c744146a-912f-4174-9c2c-e1fdedae9582-kube-api-access-tdb5m\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:09:35.970324 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:35.970284    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"
Sep 11 04:09:35.992456 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:35.992425    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"]
Sep 11 04:09:35.997788 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:35.997761    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-26814939-784f95c975-cgtc9"]
Sep 11 04:09:36.275602 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:36.275509    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c744146a-912f-4174-9c2c-e1fdedae9582" path="/var/lib/kubelet/pods/c744146a-912f-4174-9c2c-e1fdedae9582/volumes"
Sep 11 04:09:42.827805 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:42.827765    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:42.828252 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:42.827823    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:42.840445 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:42.840408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:43.020017 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:43.019988    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:09:48.192038 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:48.191999    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"]
Sep 11 04:09:58.605634 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:58.605600    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"]
Sep 11 04:09:59.291952 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.291916    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"]
Sep 11 04:09:59.430201 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.430165    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"]
Sep 11 04:09:59.430658 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.430643    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c744146a-912f-4174-9c2c-e1fdedae9582" containerName="payload-processing"
Sep 11 04:09:59.430715 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.430660    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c744146a-912f-4174-9c2c-e1fdedae9582" containerName="payload-processing"
Sep 11 04:09:59.430757 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.430747    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c744146a-912f-4174-9c2c-e1fdedae9582" containerName="payload-processing"
Sep 11 04:09:59.435267 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.435252    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"
Sep 11 04:09:59.438089 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.438065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:09:59.438192 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.438071    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:09:59.441861 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.441836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"]
Sep 11 04:09:59.548633 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.548513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44691d87-eb95-4ad7-8575-1e756f8054cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl\" (UID: \"44691d87-eb95-4ad7-8575-1e756f8054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"
Sep 11 04:09:59.649220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.649182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44691d87-eb95-4ad7-8575-1e756f8054cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl\" (UID: \"44691d87-eb95-4ad7-8575-1e756f8054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"
Sep 11 04:09:59.649853 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.649834    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44691d87-eb95-4ad7-8575-1e756f8054cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl\" (UID: \"44691d87-eb95-4ad7-8575-1e756f8054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"
Sep 11 04:09:59.748258 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:09:59.748227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"
Sep 11 04:10:00.021197 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:00.021167    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"]
Sep 11 04:10:00.023682 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:10:00.023654    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44691d87_eb95_4ad7_8575_1e756f8054cd.slice/crio-79516a2ba275f2b7f19ee7f84f90b059110aa4e5403e8fe40aa8f5c913e438a6 WatchSource:0}: Error finding container 79516a2ba275f2b7f19ee7f84f90b059110aa4e5403e8fe40aa8f5c913e438a6: Status 404 returned error can't find the container with id 79516a2ba275f2b7f19ee7f84f90b059110aa4e5403e8fe40aa8f5c913e438a6
Sep 11 04:10:00.101225 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:00.101186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl" event={"ID":"44691d87-eb95-4ad7-8575-1e756f8054cd","Type":"ContainerStarted","Data":"79516a2ba275f2b7f19ee7f84f90b059110aa4e5403e8fe40aa8f5c913e438a6"}
Sep 11 04:10:01.108109 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:01.108010    2582 generic.go:356] "Generic (PLEG): container finished" podID="44691d87-eb95-4ad7-8575-1e756f8054cd" containerID="b3de039ee033a7a636946c55959fc2ce83136ce02e8fab24d2e0515927b40624" exitCode=0
Sep 11 04:10:01.108109 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:01.108097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl" event={"ID":"44691d87-eb95-4ad7-8575-1e756f8054cd","Type":"ContainerDied","Data":"b3de039ee033a7a636946c55959fc2ce83136ce02e8fab24d2e0515927b40624"}
Sep 11 04:10:02.249734 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:02.249704    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"
Sep 11 04:10:02.275645 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:02.275619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44691d87-eb95-4ad7-8575-1e756f8054cd-maas-api-service-ca\") pod \"44691d87-eb95-4ad7-8575-1e756f8054cd\" (UID: \"44691d87-eb95-4ad7-8575-1e756f8054cd\") "
Sep 11 04:10:02.275966 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:02.275945    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44691d87-eb95-4ad7-8575-1e756f8054cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44691d87-eb95-4ad7-8575-1e756f8054cd" (UID: "44691d87-eb95-4ad7-8575-1e756f8054cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:10:02.276045 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:02.276027    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44691d87-eb95-4ad7-8575-1e756f8054cd-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:03.119314 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.119284    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"
Sep 11 04:10:03.119506 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.119323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl" event={"ID":"44691d87-eb95-4ad7-8575-1e756f8054cd","Type":"ContainerDied","Data":"79516a2ba275f2b7f19ee7f84f90b059110aa4e5403e8fe40aa8f5c913e438a6"}
Sep 11 04:10:03.119506 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.119375    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79516a2ba275f2b7f19ee7f84f90b059110aa4e5403e8fe40aa8f5c913e438a6"
Sep 11 04:10:03.211873 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.211832    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2" podUID="1464a394-b244-424f-92f4-5bc4def6a5c4" containerName="main" containerID="cri-o://b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8" gracePeriod=45
Sep 11 04:10:03.475116 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.475088    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:10:03.487641 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.487617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-tmp-dir\") pod \"1464a394-b244-424f-92f4-5bc4def6a5c4\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") "
Sep 11 04:10:03.487753 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.487652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t49bq\" (UniqueName: \"kubernetes.io/projected/1464a394-b244-424f-92f4-5bc4def6a5c4-kube-api-access-t49bq\") pod \"1464a394-b244-424f-92f4-5bc4def6a5c4\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") "
Sep 11 04:10:03.487753 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.487687    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-model-cache\") pod \"1464a394-b244-424f-92f4-5bc4def6a5c4\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") "
Sep 11 04:10:03.487753 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.487704    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-dshm\") pod \"1464a394-b244-424f-92f4-5bc4def6a5c4\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") "
Sep 11 04:10:03.487891 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.487820    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-home\") pod \"1464a394-b244-424f-92f4-5bc4def6a5c4\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") "
Sep 11 04:10:03.487891 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.487869    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1464a394-b244-424f-92f4-5bc4def6a5c4-tls-certs\") pod \"1464a394-b244-424f-92f4-5bc4def6a5c4\" (UID: \"1464a394-b244-424f-92f4-5bc4def6a5c4\") "
Sep 11 04:10:03.487998 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.487931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1464a394-b244-424f-92f4-5bc4def6a5c4" (UID: "1464a394-b244-424f-92f4-5bc4def6a5c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:03.488067 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.488011    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "1464a394-b244-424f-92f4-5bc4def6a5c4" (UID: "1464a394-b244-424f-92f4-5bc4def6a5c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:03.488125 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.488081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-home" (OuterVolumeSpecName: "home") pod "1464a394-b244-424f-92f4-5bc4def6a5c4" (UID: "1464a394-b244-424f-92f4-5bc4def6a5c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:03.488252 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.488234    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:03.488252 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.488253    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:03.488363 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.488261    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:03.490075 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.490052    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-dshm" (OuterVolumeSpecName: "dshm") pod "1464a394-b244-424f-92f4-5bc4def6a5c4" (UID: "1464a394-b244-424f-92f4-5bc4def6a5c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:03.490171 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.490075    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1464a394-b244-424f-92f4-5bc4def6a5c4-kube-api-access-t49bq" (OuterVolumeSpecName: "kube-api-access-t49bq") pod "1464a394-b244-424f-92f4-5bc4def6a5c4" (UID: "1464a394-b244-424f-92f4-5bc4def6a5c4"). InnerVolumeSpecName "kube-api-access-t49bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:03.490253 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.490233    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1464a394-b244-424f-92f4-5bc4def6a5c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1464a394-b244-424f-92f4-5bc4def6a5c4" (UID: "1464a394-b244-424f-92f4-5bc4def6a5c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:10:03.589491 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.589452    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1464a394-b244-424f-92f4-5bc4def6a5c4-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:03.589491 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.589481    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1464a394-b244-424f-92f4-5bc4def6a5c4-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:03.589491 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:03.589492    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t49bq\" (UniqueName: \"kubernetes.io/projected/1464a394-b244-424f-92f4-5bc4def6a5c4-kube-api-access-t49bq\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:04.127040 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.127000    2582 generic.go:356] "Generic (PLEG): container finished" podID="1464a394-b244-424f-92f4-5bc4def6a5c4" containerID="b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8" exitCode=0
Sep 11 04:10:04.127040 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.127040    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"
Sep 11 04:10:04.127275 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.127091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2" event={"ID":"1464a394-b244-424f-92f4-5bc4def6a5c4","Type":"ContainerDied","Data":"b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8"}
Sep 11 04:10:04.127275 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.127132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2" event={"ID":"1464a394-b244-424f-92f4-5bc4def6a5c4","Type":"ContainerDied","Data":"e7dd87ef8b5f917b64afdd285af6149d94a4cb7c253002869135be2f7c7a00f4"}
Sep 11 04:10:04.127275 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.127149    2582 scope.go:117] "RemoveContainer" containerID="b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8"
Sep 11 04:10:04.144217 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.144198    2582 scope.go:117] "RemoveContainer" containerID="b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8"
Sep 11 04:10:04.144565 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:10:04.144532    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8\": container with ID starting with b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8 not found: ID does not exist" containerID="b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8"
Sep 11 04:10:04.144623 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.144579    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8"} err="failed to get container status \"b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8\": rpc error: code = NotFound desc = could not find container \"b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8\": container with ID starting with b0e0e0d222c81bf39d3f7490d62bbe7dd445825bdb7078a6b0950831540a11f8 not found: ID does not exist"
Sep 11 04:10:04.153382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.153355    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"]
Sep 11 04:10:04.159144 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.159120    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-dc984df1-kserve-56877bcc98-9g8r2"]
Sep 11 04:10:04.275791 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.275759    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1464a394-b244-424f-92f4-5bc4def6a5c4" path="/var/lib/kubelet/pods/1464a394-b244-424f-92f4-5bc4def6a5c4/volumes"
Sep 11 04:10:04.571698 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.571661    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"]
Sep 11 04:10:04.572106 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.572099    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1464a394-b244-424f-92f4-5bc4def6a5c4" containerName="main"
Sep 11 04:10:04.572160 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.572110    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1464a394-b244-424f-92f4-5bc4def6a5c4" containerName="main"
Sep 11 04:10:04.572160 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.572118    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44691d87-eb95-4ad7-8575-1e756f8054cd" containerName="revoke-keys"
Sep 11 04:10:04.572160 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.572124    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44691d87-eb95-4ad7-8575-1e756f8054cd" containerName="revoke-keys"
Sep 11 04:10:04.572265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.572195    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1464a394-b244-424f-92f4-5bc4def6a5c4" containerName="main"
Sep 11 04:10:04.572265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.572207    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44691d87-eb95-4ad7-8575-1e756f8054cd" containerName="revoke-keys"
Sep 11 04:10:04.576817 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.576800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"
Sep 11 04:10:04.580315 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.580293    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:10:04.580315 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.580311    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:10:04.596759 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.596730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"]
Sep 11 04:10:04.600017 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.599991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074f66f-cba8-4d09-b92a-28be8fdb2e00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-285c6871-smtm4\" (UID: \"c074f66f-cba8-4d09-b92a-28be8fdb2e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"
Sep 11 04:10:04.700669 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.700624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074f66f-cba8-4d09-b92a-28be8fdb2e00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-285c6871-smtm4\" (UID: \"c074f66f-cba8-4d09-b92a-28be8fdb2e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"
Sep 11 04:10:04.701248 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.701230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074f66f-cba8-4d09-b92a-28be8fdb2e00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-285c6871-smtm4\" (UID: \"c074f66f-cba8-4d09-b92a-28be8fdb2e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"
Sep 11 04:10:04.887392 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:04.887297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"
Sep 11 04:10:05.163167 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:05.163143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"]
Sep 11 04:10:05.165096 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:10:05.165066    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc074f66f_cba8_4d09_b92a_28be8fdb2e00.slice/crio-1e857fd588aa1aef782d60ee5fb5b2babdcb1545fbcc9dcaf0920decceca8b58 WatchSource:0}: Error finding container 1e857fd588aa1aef782d60ee5fb5b2babdcb1545fbcc9dcaf0920decceca8b58: Status 404 returned error can't find the container with id 1e857fd588aa1aef782d60ee5fb5b2babdcb1545fbcc9dcaf0920decceca8b58
Sep 11 04:10:06.141488 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:06.141396    2582 generic.go:356] "Generic (PLEG): container finished" podID="c074f66f-cba8-4d09-b92a-28be8fdb2e00" containerID="7348d3129d19110be677e2464f53910941b4cf9be1e0d7a9aec976dafdad0055" exitCode=0
Sep 11 04:10:06.141488 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:06.141430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4" event={"ID":"c074f66f-cba8-4d09-b92a-28be8fdb2e00","Type":"ContainerDied","Data":"7348d3129d19110be677e2464f53910941b4cf9be1e0d7a9aec976dafdad0055"}
Sep 11 04:10:06.141488 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:06.141481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4" event={"ID":"c074f66f-cba8-4d09-b92a-28be8fdb2e00","Type":"ContainerStarted","Data":"1e857fd588aa1aef782d60ee5fb5b2babdcb1545fbcc9dcaf0920decceca8b58"}
Sep 11 04:10:07.278571 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:07.278526    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"
Sep 11 04:10:07.324693 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:07.324656    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074f66f-cba8-4d09-b92a-28be8fdb2e00-maas-api-service-ca\") pod \"c074f66f-cba8-4d09-b92a-28be8fdb2e00\" (UID: \"c074f66f-cba8-4d09-b92a-28be8fdb2e00\") "
Sep 11 04:10:07.325009 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:07.324985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c074f66f-cba8-4d09-b92a-28be8fdb2e00-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c074f66f-cba8-4d09-b92a-28be8fdb2e00" (UID: "c074f66f-cba8-4d09-b92a-28be8fdb2e00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:10:07.425875 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:07.425844    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074f66f-cba8-4d09-b92a-28be8fdb2e00-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:08.161662 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:08.161570    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"
Sep 11 04:10:08.161662 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:08.161601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4" event={"ID":"c074f66f-cba8-4d09-b92a-28be8fdb2e00","Type":"ContainerDied","Data":"1e857fd588aa1aef782d60ee5fb5b2babdcb1545fbcc9dcaf0920decceca8b58"}
Sep 11 04:10:08.161920 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:08.161672    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e857fd588aa1aef782d60ee5fb5b2babdcb1545fbcc9dcaf0920decceca8b58"
Sep 11 04:10:10.232685 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.232652    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"]
Sep 11 04:10:10.233233 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.232871    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5" podUID="71e84f2c-9997-4107-b490-c1f711b9911d" containerName="payload-processing" containerID="cri-o://73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a" gracePeriod=30
Sep 11 04:10:10.412512 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.412483    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"]
Sep 11 04:10:10.415957 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.415930    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eda9e4-cs6jl"]
Sep 11 04:10:10.494001 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.493936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:10:10.575177 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.575145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tb2q\" (UniqueName: \"kubernetes.io/projected/71e84f2c-9997-4107-b490-c1f711b9911d-kube-api-access-5tb2q\") pod \"71e84f2c-9997-4107-b490-c1f711b9911d\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") "
Sep 11 04:10:10.575366 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.575195    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e84f2c-9997-4107-b490-c1f711b9911d-plugins-config-volume\") pod \"71e84f2c-9997-4107-b490-c1f711b9911d\" (UID: \"71e84f2c-9997-4107-b490-c1f711b9911d\") "
Sep 11 04:10:10.575666 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.575645    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e84f2c-9997-4107-b490-c1f711b9911d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71e84f2c-9997-4107-b490-c1f711b9911d" (UID: "71e84f2c-9997-4107-b490-c1f711b9911d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:10:10.577317 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.577292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e84f2c-9997-4107-b490-c1f711b9911d-kube-api-access-5tb2q" (OuterVolumeSpecName: "kube-api-access-5tb2q") pod "71e84f2c-9997-4107-b490-c1f711b9911d" (UID: "71e84f2c-9997-4107-b490-c1f711b9911d"). InnerVolumeSpecName "kube-api-access-5tb2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:10.676256 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.676216    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tb2q\" (UniqueName: \"kubernetes.io/projected/71e84f2c-9997-4107-b490-c1f711b9911d-kube-api-access-5tb2q\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:10.676256 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:10.676250    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e84f2c-9997-4107-b490-c1f711b9911d-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:11.181929 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.181891    2582 generic.go:356] "Generic (PLEG): container finished" podID="71e84f2c-9997-4107-b490-c1f711b9911d" containerID="73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a" exitCode=0
Sep 11 04:10:11.182108 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.181936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"
Sep 11 04:10:11.182108 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.181945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5" event={"ID":"71e84f2c-9997-4107-b490-c1f711b9911d","Type":"ContainerDied","Data":"73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a"}
Sep 11 04:10:11.182108 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.181987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5" event={"ID":"71e84f2c-9997-4107-b490-c1f711b9911d","Type":"ContainerDied","Data":"8d1aee6f24636830c2f86fc352add0be0a50be297e9e4a7f4097986b31ad37e8"}
Sep 11 04:10:11.182108 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.182009    2582 scope.go:117] "RemoveContainer" containerID="73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a"
Sep 11 04:10:11.191300 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.191279    2582 scope.go:117] "RemoveContainer" containerID="73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a"
Sep 11 04:10:11.191587 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:10:11.191568    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a\": container with ID starting with 73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a not found: ID does not exist" containerID="73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a"
Sep 11 04:10:11.191647 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.191599    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a"} err="failed to get container status \"73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a\": rpc error: code = NotFound desc = could not find container \"73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a\": container with ID starting with 73ec702d885d284e8bbae6393f06a96fa9221c178ad32b7dac2bb7354195521a not found: ID does not exist"
Sep 11 04:10:11.203175 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.203152    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"]
Sep 11 04:10:11.207049 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:11.207024    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eda9e4-77b57bf45c-sqcz5"]
Sep 11 04:10:12.063353 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:12.063309    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"]
Sep 11 04:10:12.066417 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:12.066383    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-285c6871-smtm4"]
Sep 11 04:10:12.274704 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:12.274674    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44691d87-eb95-4ad7-8575-1e756f8054cd" path="/var/lib/kubelet/pods/44691d87-eb95-4ad7-8575-1e756f8054cd/volumes"
Sep 11 04:10:12.275070 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:12.275057    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e84f2c-9997-4107-b490-c1f711b9911d" path="/var/lib/kubelet/pods/71e84f2c-9997-4107-b490-c1f711b9911d/volumes"
Sep 11 04:10:12.275346 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:12.275335    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c074f66f-cba8-4d09-b92a-28be8fdb2e00" path="/var/lib/kubelet/pods/c074f66f-cba8-4d09-b92a-28be8fdb2e00/volumes"
Sep 11 04:10:13.624693 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:13.624634    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf" podUID="ee1dcade-791a-4dbc-9bef-de7cdfbf6997" containerName="main" containerID="cri-o://f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482" gracePeriod=45
Sep 11 04:10:13.877301 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:13.877237    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:10:14.018733 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.018696    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tls-certs\") pod \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") "
Sep 11 04:10:14.018878 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.018756    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-dshm\") pod \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") "
Sep 11 04:10:14.018878 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.018796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tmp-dir\") pod \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") "
Sep 11 04:10:14.018878 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.018865    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-home\") pod \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") "
Sep 11 04:10:14.019003 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.018894    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-model-cache\") pod \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") "
Sep 11 04:10:14.019003 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.018933    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8jqf\" (UniqueName: \"kubernetes.io/projected/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-kube-api-access-g8jqf\") pod \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\" (UID: \"ee1dcade-791a-4dbc-9bef-de7cdfbf6997\") "
Sep 11 04:10:14.019208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.019134    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee1dcade-791a-4dbc-9bef-de7cdfbf6997" (UID: "ee1dcade-791a-4dbc-9bef-de7cdfbf6997"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.019208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.019185    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-home" (OuterVolumeSpecName: "home") pod "ee1dcade-791a-4dbc-9bef-de7cdfbf6997" (UID: "ee1dcade-791a-4dbc-9bef-de7cdfbf6997"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.019208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.019196    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee1dcade-791a-4dbc-9bef-de7cdfbf6997" (UID: "ee1dcade-791a-4dbc-9bef-de7cdfbf6997"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.019453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.019379    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.019453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.019401    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.019453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.019416    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.021053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.021033    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-dshm" (OuterVolumeSpecName: "dshm") pod "ee1dcade-791a-4dbc-9bef-de7cdfbf6997" (UID: "ee1dcade-791a-4dbc-9bef-de7cdfbf6997"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.021138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.021049    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-kube-api-access-g8jqf" (OuterVolumeSpecName: "kube-api-access-g8jqf") pod "ee1dcade-791a-4dbc-9bef-de7cdfbf6997" (UID: "ee1dcade-791a-4dbc-9bef-de7cdfbf6997"). InnerVolumeSpecName "kube-api-access-g8jqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:14.021138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.021071    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee1dcade-791a-4dbc-9bef-de7cdfbf6997" (UID: "ee1dcade-791a-4dbc-9bef-de7cdfbf6997"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:10:14.120748 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.120712    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jqf\" (UniqueName: \"kubernetes.io/projected/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-kube-api-access-g8jqf\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.120748 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.120744    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.120748 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.120754    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1dcade-791a-4dbc-9bef-de7cdfbf6997-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.203074 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.203038    2582 generic.go:356] "Generic (PLEG): container finished" podID="ee1dcade-791a-4dbc-9bef-de7cdfbf6997" containerID="f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482" exitCode=0
Sep 11 04:10:14.203287 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.203080    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"
Sep 11 04:10:14.203287 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.203119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf" event={"ID":"ee1dcade-791a-4dbc-9bef-de7cdfbf6997","Type":"ContainerDied","Data":"f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482"}
Sep 11 04:10:14.203287 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.203174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf" event={"ID":"ee1dcade-791a-4dbc-9bef-de7cdfbf6997","Type":"ContainerDied","Data":"6421a466bb2d9f4da739e59185c61030bf64d1b8914562d216a57cff2086931b"}
Sep 11 04:10:14.203287 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.203196    2582 scope.go:117] "RemoveContainer" containerID="f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482"
Sep 11 04:10:14.212504 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.212484    2582 scope.go:117] "RemoveContainer" containerID="f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482"
Sep 11 04:10:14.212782 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:10:14.212764    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482\": container with ID starting with f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482 not found: ID does not exist" containerID="f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482"
Sep 11 04:10:14.212849 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.212794    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482"} err="failed to get container status \"f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482\": rpc error: code = NotFound desc = could not find container \"f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482\": container with ID starting with f14c58aaeca5625227109f9db60bd33df7b2a2271490715bdeb758db67a76482 not found: ID does not exist"
Sep 11 04:10:14.224864 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.224837    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"]
Sep 11 04:10:14.228439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.228415    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-dc984df1-kserve-85c74d8988-dc2jf"]
Sep 11 04:10:14.275051 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.275019    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1dcade-791a-4dbc-9bef-de7cdfbf6997" path="/var/lib/kubelet/pods/ee1dcade-791a-4dbc-9bef-de7cdfbf6997/volumes"
Sep 11 04:10:14.312194 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.312154    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw" podUID="173797d7-69ae-48e5-acad-47e36d9909b6" containerName="main" containerID="cri-o://014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059" gracePeriod=45
Sep 11 04:10:14.543021 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.542998    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:10:14.625118 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-model-cache\") pod \"173797d7-69ae-48e5-acad-47e36d9909b6\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") "
Sep 11 04:10:14.625118 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625121    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-dshm\") pod \"173797d7-69ae-48e5-acad-47e36d9909b6\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") "
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj6pn\" (UniqueName: \"kubernetes.io/projected/173797d7-69ae-48e5-acad-47e36d9909b6-kube-api-access-fj6pn\") pod \"173797d7-69ae-48e5-acad-47e36d9909b6\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") "
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-home\") pod \"173797d7-69ae-48e5-acad-47e36d9909b6\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") "
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173797d7-69ae-48e5-acad-47e36d9909b6-tls-certs\") pod \"173797d7-69ae-48e5-acad-47e36d9909b6\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") "
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-tmp-dir\") pod \"173797d7-69ae-48e5-acad-47e36d9909b6\" (UID: \"173797d7-69ae-48e5-acad-47e36d9909b6\") "
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625400    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "173797d7-69ae-48e5-acad-47e36d9909b6" (UID: "173797d7-69ae-48e5-acad-47e36d9909b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625500    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-home" (OuterVolumeSpecName: "home") pod "173797d7-69ae-48e5-acad-47e36d9909b6" (UID: "173797d7-69ae-48e5-acad-47e36d9909b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "173797d7-69ae-48e5-acad-47e36d9909b6" (UID: "173797d7-69ae-48e5-acad-47e36d9909b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.625675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625677    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.625976 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.625690    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.627605 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.627583    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173797d7-69ae-48e5-acad-47e36d9909b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "173797d7-69ae-48e5-acad-47e36d9909b6" (UID: "173797d7-69ae-48e5-acad-47e36d9909b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:10:14.627733 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.627625    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173797d7-69ae-48e5-acad-47e36d9909b6-kube-api-access-fj6pn" (OuterVolumeSpecName: "kube-api-access-fj6pn") pod "173797d7-69ae-48e5-acad-47e36d9909b6" (UID: "173797d7-69ae-48e5-acad-47e36d9909b6"). InnerVolumeSpecName "kube-api-access-fj6pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:14.627733 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.627632    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-dshm" (OuterVolumeSpecName: "dshm") pod "173797d7-69ae-48e5-acad-47e36d9909b6" (UID: "173797d7-69ae-48e5-acad-47e36d9909b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:14.727177 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.727142    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.727177 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.727173    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj6pn\" (UniqueName: \"kubernetes.io/projected/173797d7-69ae-48e5-acad-47e36d9909b6-kube-api-access-fj6pn\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.727177 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.727185    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173797d7-69ae-48e5-acad-47e36d9909b6-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:14.727415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:14.727194    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173797d7-69ae-48e5-acad-47e36d9909b6-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:10:15.209343 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.209306    2582 generic.go:356] "Generic (PLEG): container finished" podID="173797d7-69ae-48e5-acad-47e36d9909b6" containerID="014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059" exitCode=0
Sep 11 04:10:15.209524 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.209357    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"
Sep 11 04:10:15.209524 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.209394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw" event={"ID":"173797d7-69ae-48e5-acad-47e36d9909b6","Type":"ContainerDied","Data":"014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059"}
Sep 11 04:10:15.209524 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.209436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw" event={"ID":"173797d7-69ae-48e5-acad-47e36d9909b6","Type":"ContainerDied","Data":"e9d529f2aefedf39bc227f0e9c6a3d2e5bdc9e707fc46bf1f223635caab7cef0"}
Sep 11 04:10:15.209524 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.209454    2582 scope.go:117] "RemoveContainer" containerID="014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059"
Sep 11 04:10:15.219797 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.219780    2582 scope.go:117] "RemoveContainer" containerID="014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059"
Sep 11 04:10:15.220057 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:10:15.220038    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059\": container with ID starting with 014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059 not found: ID does not exist" containerID="014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059"
Sep 11 04:10:15.220115 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.220066    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059"} err="failed to get container status \"014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059\": rpc error: code = NotFound desc = could not find container \"014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059\": container with ID starting with 014c058441829f6ffc8c600f02128c6e1c9d9ed67cd200a9843f55766a99d059 not found: ID does not exist"
Sep 11 04:10:15.232269 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.232242    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"]
Sep 11 04:10:15.237824 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:15.237803    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-eda9e4/autores-model-eda9e4-kserve-86bfb765bc-dkzbw"]
Sep 11 04:10:16.276818 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:16.276782    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173797d7-69ae-48e5-acad-47e36d9909b6" path="/var/lib/kubelet/pods/173797d7-69ae-48e5-acad-47e36d9909b6/volumes"
Sep 11 04:10:22.760451 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760416    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"]
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760860    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1dcade-791a-4dbc-9bef-de7cdfbf6997" containerName="main"
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760873    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1dcade-791a-4dbc-9bef-de7cdfbf6997" containerName="main"
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760888    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173797d7-69ae-48e5-acad-47e36d9909b6" containerName="main"
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760894    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="173797d7-69ae-48e5-acad-47e36d9909b6" containerName="main"
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760901    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c074f66f-cba8-4d09-b92a-28be8fdb2e00" containerName="revoke-keys"
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760907    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c074f66f-cba8-4d09-b92a-28be8fdb2e00" containerName="revoke-keys"
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760918    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e84f2c-9997-4107-b490-c1f711b9911d" containerName="payload-processing"
Sep 11 04:10:22.760941 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760924    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e84f2c-9997-4107-b490-c1f711b9911d" containerName="payload-processing"
Sep 11 04:10:22.761206 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760984    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e84f2c-9997-4107-b490-c1f711b9911d" containerName="payload-processing"
Sep 11 04:10:22.761206 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.760993    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="173797d7-69ae-48e5-acad-47e36d9909b6" containerName="main"
Sep 11 04:10:22.761206 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.761003    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c074f66f-cba8-4d09-b92a-28be8fdb2e00" containerName="revoke-keys"
Sep 11 04:10:22.761206 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.761010    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1dcade-791a-4dbc-9bef-de7cdfbf6997" containerName="main"
Sep 11 04:10:22.765705 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.765688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:22.768365 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.768343    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-57c530-dockercfg-2q7s2\""
Sep 11 04:10:22.768510 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.768493    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-57c530\""
Sep 11 04:10:22.785151 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.785125    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"]
Sep 11 04:10:22.910550 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.910515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:22.910751 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:22.910605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qvg\" (UniqueName: \"kubernetes.io/projected/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-kube-api-access-28qvg\") pod \"payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:23.011958 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:23.011868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:23.011958 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:23.011930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qvg\" (UniqueName: \"kubernetes.io/projected/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-kube-api-access-28qvg\") pod \"payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:23.012598 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:23.012542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:23.021716 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:23.021690    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qvg\" (UniqueName: \"kubernetes.io/projected/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-kube-api-access-28qvg\") pod \"payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:23.076037 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:23.075999    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:23.357751 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:23.357727    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"]
Sep 11 04:10:23.360295 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:10:23.360137    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ca7462_1638_4433_b6ba_f9f32d3b2d65.slice/crio-1844070f8819698132b6c046df98258bb10b96784140de230954e68ad394fab2 WatchSource:0}: Error finding container 1844070f8819698132b6c046df98258bb10b96784140de230954e68ad394fab2: Status 404 returned error can't find the container with id 1844070f8819698132b6c046df98258bb10b96784140de230954e68ad394fab2
Sep 11 04:10:24.261842 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:24.261807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98" event={"ID":"e4ca7462-1638-4433-b6ba-f9f32d3b2d65","Type":"ContainerStarted","Data":"0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35"}
Sep 11 04:10:24.262221 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:24.261864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98" event={"ID":"e4ca7462-1638-4433-b6ba-f9f32d3b2d65","Type":"ContainerStarted","Data":"1844070f8819698132b6c046df98258bb10b96784140de230954e68ad394fab2"}
Sep 11 04:10:24.262221 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:24.261891    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:24.283803 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:24.283746    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98" podStartSLOduration=2.2837261939999998 podStartE2EDuration="2.283726194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:10:24.283221639 +0000 UTC m=+998.547700013" watchObservedRunningTime="2026-09-11 04:10:24.283726194 +0000 UTC m=+998.548204567"
Sep 11 04:10:27.107767 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.107723    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"]
Sep 11 04:10:27.111656 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.111636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.114805 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.114786    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fca08f\"/\"default-dockercfg-47ms8\""
Sep 11 04:10:27.115880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.115858    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fca08f\"/\"kube-root-ca.crt\""
Sep 11 04:10:27.116133 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.116116    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fca08f\"/\"test-model-fca08f-kserve-self-signed-certs\""
Sep 11 04:10:27.116936 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.116917    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fca08f\"/\"openshift-service-ca.crt\""
Sep 11 04:10:27.122520 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.122500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"]
Sep 11 04:10:27.154457 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.154423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn729\" (UniqueName: \"kubernetes.io/projected/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-kube-api-access-rn729\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.154457 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.154458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-dshm\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.154716 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.154484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-model-cache\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.154716 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.154616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tls-certs\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.154716 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.154700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tmp-dir\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.154871 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.154743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-home\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.255792 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.255755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tls-certs\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.255982 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.255822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tmp-dir\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.255982 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.255845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-home\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.255982 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.255925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn729\" (UniqueName: \"kubernetes.io/projected/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-kube-api-access-rn729\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.255982 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.255954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-dshm\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.256193 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.255989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-model-cache\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.256291 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.256265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tmp-dir\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.256354 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.256294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-home\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.256425 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.256405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-model-cache\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.258112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.258082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-dshm\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.258438 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.258419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tls-certs\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.264013 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.263990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn729\" (UniqueName: \"kubernetes.io/projected/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-kube-api-access-rn729\") pod \"test-model-fca08f-kserve-55bcc6456f-dnhx7\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") " pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.421685 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.421648    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:27.696601 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:27.696574    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"]
Sep 11 04:10:27.698963 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:10:27.698935    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod194a5ba5_a340_46b8_b11e_5b15ab0c25cd.slice/crio-6cd3b6dbbcb4b64ed3662b55fdac833882422399a7a68067189801053fb901c6 WatchSource:0}: Error finding container 6cd3b6dbbcb4b64ed3662b55fdac833882422399a7a68067189801053fb901c6: Status 404 returned error can't find the container with id 6cd3b6dbbcb4b64ed3662b55fdac833882422399a7a68067189801053fb901c6
Sep 11 04:10:28.285874 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:28.285829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7" event={"ID":"194a5ba5-a340-46b8-b11e-5b15ab0c25cd","Type":"ContainerStarted","Data":"e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5"}
Sep 11 04:10:28.285874 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:28.285881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7" event={"ID":"194a5ba5-a340-46b8-b11e-5b15ab0c25cd","Type":"ContainerStarted","Data":"6cd3b6dbbcb4b64ed3662b55fdac833882422399a7a68067189801053fb901c6"}
Sep 11 04:10:28.305389 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:28.305331    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7" podStartSLOduration=1.305315095 podStartE2EDuration="1.305315095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:10:28.303177221 +0000 UTC m=+1002.567655587" watchObservedRunningTime="2026-09-11 04:10:28.305315095 +0000 UTC m=+1002.569793466"
Sep 11 04:10:35.269885 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:35.269845    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:10:37.422293 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:37.422250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:37.422817 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:37.422310    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:37.435662 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:37.435629    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:38.354990 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:38.354960    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:10:58.510537 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.510504    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"]
Sep 11 04:10:58.514241 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.514221    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.520067 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.520041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-57c530\"/\"kube-root-ca.crt\""
Sep 11 04:10:58.520239 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.520066    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-57c530\"/\"explicit-model-57c530-kserve-self-signed-certs\""
Sep 11 04:10:58.520239 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.520066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-57c530\"/\"openshift-service-ca.crt\""
Sep 11 04:10:58.520239 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.520066    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-57c530\"/\"default-dockercfg-gp7xp\""
Sep 11 04:10:58.534753 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.534723    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"]
Sep 11 04:10:58.679954 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.679917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-model-cache\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.680155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.679995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgx64\" (UniqueName: \"kubernetes.io/projected/67bb3baf-4a17-4c30-acda-22079ac2601e-kube-api-access-qgx64\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.680155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.680026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bb3baf-4a17-4c30-acda-22079ac2601e-tls-certs\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.680155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.680093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-tmp-dir\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.680155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.680131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-dshm\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.680330 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.680165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-home\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.781725 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.781636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgx64\" (UniqueName: \"kubernetes.io/projected/67bb3baf-4a17-4c30-acda-22079ac2601e-kube-api-access-qgx64\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.781725 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.781677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bb3baf-4a17-4c30-acda-22079ac2601e-tls-certs\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.781725 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.781713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-tmp-dir\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.781979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.781877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-dshm\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.781979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.781931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-home\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.782087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.782028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-model-cache\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.782145 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.782130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-tmp-dir\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.782283 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.782265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-home\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.782403 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.782375    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-model-cache\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.784157 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.784136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-dshm\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.784330 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.784312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bb3baf-4a17-4c30-acda-22079ac2601e-tls-certs\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.790589 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.790547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgx64\" (UniqueName: \"kubernetes.io/projected/67bb3baf-4a17-4c30-acda-22079ac2601e-kube-api-access-qgx64\") pod \"explicit-model-57c530-kserve-6c85c76475-fvrc4\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") " pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:58.823452 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:58.823417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:10:59.099188 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:59.099160    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"]
Sep 11 04:10:59.101653 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:10:59.101607    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67bb3baf_4a17_4c30_acda_22079ac2601e.slice/crio-1ad0e46f57be569b64e0b966b93961d586be2b44b7b5b29c264ffac18e3e1661 WatchSource:0}: Error finding container 1ad0e46f57be569b64e0b966b93961d586be2b44b7b5b29c264ffac18e3e1661: Status 404 returned error can't find the container with id 1ad0e46f57be569b64e0b966b93961d586be2b44b7b5b29c264ffac18e3e1661
Sep 11 04:10:59.466208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:59.466173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4" event={"ID":"67bb3baf-4a17-4c30-acda-22079ac2601e","Type":"ContainerStarted","Data":"13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8"}
Sep 11 04:10:59.466208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:59.466213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4" event={"ID":"67bb3baf-4a17-4c30-acda-22079ac2601e","Type":"ContainerStarted","Data":"1ad0e46f57be569b64e0b966b93961d586be2b44b7b5b29c264ffac18e3e1661"}
Sep 11 04:10:59.486108 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:10:59.486052    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4" podStartSLOduration=1.486035513 podStartE2EDuration="1.486035513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:10:59.484113888 +0000 UTC m=+1033.748592259" watchObservedRunningTime="2026-09-11 04:10:59.486035513 +0000 UTC m=+1033.750513883"
Sep 11 04:11:08.823679 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:08.823641    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:11:08.824099 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:08.823693    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:11:08.836605 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:08.836570    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:11:09.536820 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:09.536787    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:11:22.182818 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:22.182776    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"]
Sep 11 04:11:32.882472 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:32.882438    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"]
Sep 11 04:11:32.883106 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:32.882694    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98" podUID="e4ca7462-1638-4433-b6ba-f9f32d3b2d65" containerName="payload-processing" containerID="cri-o://0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35" gracePeriod=30
Sep 11 04:11:33.133496 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.133429    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:11:33.234545 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.234509    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-plugins-config-volume\") pod \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") "
Sep 11 04:11:33.234735 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.234617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28qvg\" (UniqueName: \"kubernetes.io/projected/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-kube-api-access-28qvg\") pod \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\" (UID: \"e4ca7462-1638-4433-b6ba-f9f32d3b2d65\") "
Sep 11 04:11:33.234900 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.234877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4ca7462-1638-4433-b6ba-f9f32d3b2d65" (UID: "e4ca7462-1638-4433-b6ba-f9f32d3b2d65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:11:33.235014 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.234996    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:33.236754 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.236728    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-kube-api-access-28qvg" (OuterVolumeSpecName: "kube-api-access-28qvg") pod "e4ca7462-1638-4433-b6ba-f9f32d3b2d65" (UID: "e4ca7462-1638-4433-b6ba-f9f32d3b2d65"). InnerVolumeSpecName "kube-api-access-28qvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:11:33.336322 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.336285    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28qvg\" (UniqueName: \"kubernetes.io/projected/e4ca7462-1638-4433-b6ba-f9f32d3b2d65-kube-api-access-28qvg\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:33.656894 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.656856    2582 generic.go:356] "Generic (PLEG): container finished" podID="e4ca7462-1638-4433-b6ba-f9f32d3b2d65" containerID="0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35" exitCode=0
Sep 11 04:11:33.657073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.656897    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"
Sep 11 04:11:33.657073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.656945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98" event={"ID":"e4ca7462-1638-4433-b6ba-f9f32d3b2d65","Type":"ContainerDied","Data":"0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35"}
Sep 11 04:11:33.657073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.657001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98" event={"ID":"e4ca7462-1638-4433-b6ba-f9f32d3b2d65","Type":"ContainerDied","Data":"1844070f8819698132b6c046df98258bb10b96784140de230954e68ad394fab2"}
Sep 11 04:11:33.657073 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.657028    2582 scope.go:117] "RemoveContainer" containerID="0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35"
Sep 11 04:11:33.667071 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.667049    2582 scope.go:117] "RemoveContainer" containerID="0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35"
Sep 11 04:11:33.667351 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:11:33.667334    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35\": container with ID starting with 0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35 not found: ID does not exist" containerID="0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35"
Sep 11 04:11:33.667403 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.667357    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35"} err="failed to get container status \"0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35\": rpc error: code = NotFound desc = could not find container \"0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35\": container with ID starting with 0bda5a2ee2b134aac96bb78a9b68ba899a79b534b0739e46b582ced40842ad35 not found: ID does not exist"
Sep 11 04:11:33.680400 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.680348    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"]
Sep 11 04:11:33.682658 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:33.682634    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c530-6695dfdc4f-2ch98"]
Sep 11 04:11:34.275618 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:34.275581    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ca7462-1638-4433-b6ba-f9f32d3b2d65" path="/var/lib/kubelet/pods/e4ca7462-1638-4433-b6ba-f9f32d3b2d65/volumes"
Sep 11 04:11:37.203341 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.203298    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4" podUID="67bb3baf-4a17-4c30-acda-22079ac2601e" containerName="main" containerID="cri-o://13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8" gracePeriod=45
Sep 11 04:11:37.455237 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.455171    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:11:37.476443 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-dshm\") pod \"67bb3baf-4a17-4c30-acda-22079ac2601e\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") "
Sep 11 04:11:37.476632 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-home\") pod \"67bb3baf-4a17-4c30-acda-22079ac2601e\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") "
Sep 11 04:11:37.476632 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476524    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bb3baf-4a17-4c30-acda-22079ac2601e-tls-certs\") pod \"67bb3baf-4a17-4c30-acda-22079ac2601e\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") "
Sep 11 04:11:37.476779 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476643    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgx64\" (UniqueName: \"kubernetes.io/projected/67bb3baf-4a17-4c30-acda-22079ac2601e-kube-api-access-qgx64\") pod \"67bb3baf-4a17-4c30-acda-22079ac2601e\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") "
Sep 11 04:11:37.476779 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476679    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-tmp-dir\") pod \"67bb3baf-4a17-4c30-acda-22079ac2601e\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") "
Sep 11 04:11:37.476779 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-model-cache\") pod \"67bb3baf-4a17-4c30-acda-22079ac2601e\" (UID: \"67bb3baf-4a17-4c30-acda-22079ac2601e\") "
Sep 11 04:11:37.476932 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476760    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-home" (OuterVolumeSpecName: "home") pod "67bb3baf-4a17-4c30-acda-22079ac2601e" (UID: "67bb3baf-4a17-4c30-acda-22079ac2601e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:37.476932 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.476922    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67bb3baf-4a17-4c30-acda-22079ac2601e" (UID: "67bb3baf-4a17-4c30-acda-22079ac2601e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:37.477081 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.477058    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:37.477148 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.477086    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:37.477148 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.477104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-model-cache" (OuterVolumeSpecName: "model-cache") pod "67bb3baf-4a17-4c30-acda-22079ac2601e" (UID: "67bb3baf-4a17-4c30-acda-22079ac2601e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:37.478908 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.478880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-dshm" (OuterVolumeSpecName: "dshm") pod "67bb3baf-4a17-4c30-acda-22079ac2601e" (UID: "67bb3baf-4a17-4c30-acda-22079ac2601e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:37.478908 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.478894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bb3baf-4a17-4c30-acda-22079ac2601e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67bb3baf-4a17-4c30-acda-22079ac2601e" (UID: "67bb3baf-4a17-4c30-acda-22079ac2601e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:11:37.479050 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.478986    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bb3baf-4a17-4c30-acda-22079ac2601e-kube-api-access-qgx64" (OuterVolumeSpecName: "kube-api-access-qgx64") pod "67bb3baf-4a17-4c30-acda-22079ac2601e" (UID: "67bb3baf-4a17-4c30-acda-22079ac2601e"). InnerVolumeSpecName "kube-api-access-qgx64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:11:37.577787 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.577737    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgx64\" (UniqueName: \"kubernetes.io/projected/67bb3baf-4a17-4c30-acda-22079ac2601e-kube-api-access-qgx64\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:37.577787 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.577779    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:37.577787 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.577790    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bb3baf-4a17-4c30-acda-22079ac2601e-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:37.577787 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.577797    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bb3baf-4a17-4c30-acda-22079ac2601e-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:37.681425 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.681389    2582 generic.go:356] "Generic (PLEG): container finished" podID="67bb3baf-4a17-4c30-acda-22079ac2601e" containerID="13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8" exitCode=0
Sep 11 04:11:37.681623 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.681434    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"
Sep 11 04:11:37.681623 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.681461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4" event={"ID":"67bb3baf-4a17-4c30-acda-22079ac2601e","Type":"ContainerDied","Data":"13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8"}
Sep 11 04:11:37.681623 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.681498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4" event={"ID":"67bb3baf-4a17-4c30-acda-22079ac2601e","Type":"ContainerDied","Data":"1ad0e46f57be569b64e0b966b93961d586be2b44b7b5b29c264ffac18e3e1661"}
Sep 11 04:11:37.681623 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.681520    2582 scope.go:117] "RemoveContainer" containerID="13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8"
Sep 11 04:11:37.691650 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.691628    2582 scope.go:117] "RemoveContainer" containerID="13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8"
Sep 11 04:11:37.691958 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:11:37.691938    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8\": container with ID starting with 13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8 not found: ID does not exist" containerID="13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8"
Sep 11 04:11:37.692006 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.691968    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8"} err="failed to get container status \"13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8\": rpc error: code = NotFound desc = could not find container \"13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8\": container with ID starting with 13015c39e46cb82f1ddded0b7e03b693519aa274ed91289cdaa6b23ac6a437f8 not found: ID does not exist"
Sep 11 04:11:37.703529 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.703497    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"]
Sep 11 04:11:37.707883 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:37.707824    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-57c530/explicit-model-57c530-kserve-6c85c76475-fvrc4"]
Sep 11 04:11:38.275105 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:38.275074    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67bb3baf-4a17-4c30-acda-22079ac2601e" path="/var/lib/kubelet/pods/67bb3baf-4a17-4c30-acda-22079ac2601e/volumes"
Sep 11 04:11:42.143661 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.143612    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"]
Sep 11 04:11:42.286483 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.286449    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"]
Sep 11 04:11:42.286927 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.286911    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ca7462-1638-4433-b6ba-f9f32d3b2d65" containerName="payload-processing"
Sep 11 04:11:42.286976 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.286930    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ca7462-1638-4433-b6ba-f9f32d3b2d65" containerName="payload-processing"
Sep 11 04:11:42.286976 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.286949    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bb3baf-4a17-4c30-acda-22079ac2601e" containerName="main"
Sep 11 04:11:42.286976 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.286955    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bb3baf-4a17-4c30-acda-22079ac2601e" containerName="main"
Sep 11 04:11:42.287095 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.287024    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ca7462-1638-4433-b6ba-f9f32d3b2d65" containerName="payload-processing"
Sep 11 04:11:42.287095 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.287032    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bb3baf-4a17-4c30-acda-22079ac2601e" containerName="main"
Sep 11 04:11:42.291295 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.291277    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"
Sep 11 04:11:42.294397 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.294379    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:11:42.294567 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.294536    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:11:42.321361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.321304    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"]
Sep 11 04:11:42.337239 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.337198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e165c621-e9aa-4b73-b6ba-3c589ffda3ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp\" (UID: \"e165c621-e9aa-4b73-b6ba-3c589ffda3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"
Sep 11 04:11:42.437940 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.437905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e165c621-e9aa-4b73-b6ba-3c589ffda3ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp\" (UID: \"e165c621-e9aa-4b73-b6ba-3c589ffda3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"
Sep 11 04:11:42.438848 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.438816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e165c621-e9aa-4b73-b6ba-3c589ffda3ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp\" (UID: \"e165c621-e9aa-4b73-b6ba-3c589ffda3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"
Sep 11 04:11:42.601700 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.601652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"
Sep 11 04:11:42.878837 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:42.878814    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"]
Sep 11 04:11:42.882039 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:11:42.881999    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode165c621_e9aa_4b73_b6ba_3c589ffda3ea.slice/crio-f373f04be9333649a9947395e35e6cebca7bd6d5cbb9f7613f4225ef3ab4e84b WatchSource:0}: Error finding container f373f04be9333649a9947395e35e6cebca7bd6d5cbb9f7613f4225ef3ab4e84b: Status 404 returned error can't find the container with id f373f04be9333649a9947395e35e6cebca7bd6d5cbb9f7613f4225ef3ab4e84b
Sep 11 04:11:43.719430 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:43.719398    2582 generic.go:356] "Generic (PLEG): container finished" podID="e165c621-e9aa-4b73-b6ba-3c589ffda3ea" containerID="819f4270ce5c45de2d98718d843ee750aed044a9544fcacea6a177464a715b5e" exitCode=0
Sep 11 04:11:43.719897 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:43.719451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp" event={"ID":"e165c621-e9aa-4b73-b6ba-3c589ffda3ea","Type":"ContainerDied","Data":"819f4270ce5c45de2d98718d843ee750aed044a9544fcacea6a177464a715b5e"}
Sep 11 04:11:43.719897 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:43.719501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp" event={"ID":"e165c621-e9aa-4b73-b6ba-3c589ffda3ea","Type":"ContainerStarted","Data":"f373f04be9333649a9947395e35e6cebca7bd6d5cbb9f7613f4225ef3ab4e84b"}
Sep 11 04:11:44.859353 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:44.859328    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"
Sep 11 04:11:44.960188 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:44.960153    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e165c621-e9aa-4b73-b6ba-3c589ffda3ea-maas-api-service-ca\") pod \"e165c621-e9aa-4b73-b6ba-3c589ffda3ea\" (UID: \"e165c621-e9aa-4b73-b6ba-3c589ffda3ea\") "
Sep 11 04:11:44.960524 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:44.960502    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e165c621-e9aa-4b73-b6ba-3c589ffda3ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e165c621-e9aa-4b73-b6ba-3c589ffda3ea" (UID: "e165c621-e9aa-4b73-b6ba-3c589ffda3ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:11:45.061179 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:45.061095    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e165c621-e9aa-4b73-b6ba-3c589ffda3ea-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:45.732213 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:45.732177    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"
Sep 11 04:11:45.732213 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:45.732206    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp" event={"ID":"e165c621-e9aa-4b73-b6ba-3c589ffda3ea","Type":"ContainerDied","Data":"f373f04be9333649a9947395e35e6cebca7bd6d5cbb9f7613f4225ef3ab4e84b"}
Sep 11 04:11:45.732447 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:45.732257    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f373f04be9333649a9947395e35e6cebca7bd6d5cbb9f7613f4225ef3ab4e84b"
Sep 11 04:11:52.828328 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:52.828289    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"]
Sep 11 04:11:52.828747 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:52.828521    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287" podUID="d86388a0-e58f-459d-a785-1e414f79083c" containerName="payload-pre-processing" containerID="cri-o://a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34" gracePeriod=30
Sep 11 04:11:52.988823 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:52.988791    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"]
Sep 11 04:11:52.990820 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:52.990793    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a28c9f-5w7wp"]
Sep 11 04:11:53.086493 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.086434    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:11:53.244391 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.244357    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86388a0-e58f-459d-a785-1e414f79083c-plugins-config-volume\") pod \"d86388a0-e58f-459d-a785-1e414f79083c\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") "
Sep 11 04:11:53.244598 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.244470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgcbt\" (UniqueName: \"kubernetes.io/projected/d86388a0-e58f-459d-a785-1e414f79083c-kube-api-access-pgcbt\") pod \"d86388a0-e58f-459d-a785-1e414f79083c\" (UID: \"d86388a0-e58f-459d-a785-1e414f79083c\") "
Sep 11 04:11:53.244883 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.244859    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86388a0-e58f-459d-a785-1e414f79083c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d86388a0-e58f-459d-a785-1e414f79083c" (UID: "d86388a0-e58f-459d-a785-1e414f79083c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:11:53.246670 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.246639    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86388a0-e58f-459d-a785-1e414f79083c-kube-api-access-pgcbt" (OuterVolumeSpecName: "kube-api-access-pgcbt") pod "d86388a0-e58f-459d-a785-1e414f79083c" (UID: "d86388a0-e58f-459d-a785-1e414f79083c"). InnerVolumeSpecName "kube-api-access-pgcbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:11:53.345613 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.345514    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86388a0-e58f-459d-a785-1e414f79083c-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:53.345613 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.345541    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgcbt\" (UniqueName: \"kubernetes.io/projected/d86388a0-e58f-459d-a785-1e414f79083c-kube-api-access-pgcbt\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:53.779687 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.779651    2582 generic.go:356] "Generic (PLEG): container finished" podID="d86388a0-e58f-459d-a785-1e414f79083c" containerID="a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34" exitCode=0
Sep 11 04:11:53.779859 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.779695    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"
Sep 11 04:11:53.779859 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.779737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287" event={"ID":"d86388a0-e58f-459d-a785-1e414f79083c","Type":"ContainerDied","Data":"a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34"}
Sep 11 04:11:53.779859 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.779784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287" event={"ID":"d86388a0-e58f-459d-a785-1e414f79083c","Type":"ContainerDied","Data":"1891d0f3eaa265e9b0f8284479d5397a3435b2658c4b7289650ef10c18ce8044"}
Sep 11 04:11:53.779859 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.779806    2582 scope.go:117] "RemoveContainer" containerID="a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34"
Sep 11 04:11:53.788857 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.788839    2582 scope.go:117] "RemoveContainer" containerID="a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34"
Sep 11 04:11:53.789116 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:11:53.789099    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34\": container with ID starting with a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34 not found: ID does not exist" containerID="a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34"
Sep 11 04:11:53.789158 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.789130    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34"} err="failed to get container status \"a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34\": rpc error: code = NotFound desc = could not find container \"a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34\": container with ID starting with a106801dca2c536a5576b32c596685bd2933c9ec2f79d2644be6f8fc631deb34 not found: ID does not exist"
Sep 11 04:11:53.809373 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.809345    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"]
Sep 11 04:11:53.820684 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:53.820661    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a28c9f-5f66dfcbf4-cd287"]
Sep 11 04:11:54.274801 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:54.274770    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86388a0-e58f-459d-a785-1e414f79083c" path="/var/lib/kubelet/pods/d86388a0-e58f-459d-a785-1e414f79083c/volumes"
Sep 11 04:11:54.275220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:54.275197    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e165c621-e9aa-4b73-b6ba-3c589ffda3ea" path="/var/lib/kubelet/pods/e165c621-e9aa-4b73-b6ba-3c589ffda3ea/volumes"
Sep 11 04:11:55.333791 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.333735    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"]
Sep 11 04:11:55.334229 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.334213    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e165c621-e9aa-4b73-b6ba-3c589ffda3ea" containerName="revoke-keys"
Sep 11 04:11:55.334288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.334230    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e165c621-e9aa-4b73-b6ba-3c589ffda3ea" containerName="revoke-keys"
Sep 11 04:11:55.334288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.334242    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86388a0-e58f-459d-a785-1e414f79083c" containerName="payload-pre-processing"
Sep 11 04:11:55.334288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.334247    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86388a0-e58f-459d-a785-1e414f79083c" containerName="payload-pre-processing"
Sep 11 04:11:55.334381 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.334314    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86388a0-e58f-459d-a785-1e414f79083c" containerName="payload-pre-processing"
Sep 11 04:11:55.334381 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.334327    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e165c621-e9aa-4b73-b6ba-3c589ffda3ea" containerName="revoke-keys"
Sep 11 04:11:55.338744 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.338720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.341366 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.341332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-34523cde-dockercfg-7bngv\""
Sep 11 04:11:55.341526 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.341397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-34523cde\""
Sep 11 04:11:55.349158 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.349130    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"]
Sep 11 04:11:55.467730 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.467686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lw29\" (UniqueName: \"kubernetes.io/projected/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-kube-api-access-9lw29\") pod \"payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") " pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.467888 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.467740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") " pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.568671 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.568629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lw29\" (UniqueName: \"kubernetes.io/projected/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-kube-api-access-9lw29\") pod \"payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") " pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.568841 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.568678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") " pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.569439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.569414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") " pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.576820 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.576794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lw29\" (UniqueName: \"kubernetes.io/projected/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-kube-api-access-9lw29\") pod \"payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") " pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.652323 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.652224    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:55.934171 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:55.934143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"]
Sep 11 04:11:55.936418 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:11:55.936392    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197b0592_9e96_4fc8_8eae_3d38fcdda0ef.slice/crio-b2d00dfc5ce8f00cbccbf147a2f1e343391de448f580d12c020ff5e070205d25 WatchSource:0}: Error finding container b2d00dfc5ce8f00cbccbf147a2f1e343391de448f580d12c020ff5e070205d25: Status 404 returned error can't find the container with id b2d00dfc5ce8f00cbccbf147a2f1e343391de448f580d12c020ff5e070205d25
Sep 11 04:11:56.800590 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:56.800536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv" event={"ID":"197b0592-9e96-4fc8-8eae-3d38fcdda0ef","Type":"ContainerStarted","Data":"60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8"}
Sep 11 04:11:56.800963 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:56.800610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv" event={"ID":"197b0592-9e96-4fc8-8eae-3d38fcdda0ef","Type":"ContainerStarted","Data":"b2d00dfc5ce8f00cbccbf147a2f1e343391de448f580d12c020ff5e070205d25"}
Sep 11 04:11:56.800963 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:56.800653    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:11:56.818896 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:56.818838    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv" podStartSLOduration=1.818818697 podStartE2EDuration="1.818818697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:56.817401404 +0000 UTC m=+1091.081879795" watchObservedRunningTime="2026-09-11 04:11:56.818818697 +0000 UTC m=+1091.083297068"
Sep 11 04:11:57.163511 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.163419    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7" podUID="194a5ba5-a340-46b8-b11e-5b15ab0c25cd" containerName="main" containerID="cri-o://e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5" gracePeriod=45
Sep 11 04:11:57.425303 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.425278    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:11:57.594634 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.594587    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tls-certs\") pod \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") "
Sep 11 04:11:57.594825 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.594664    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn729\" (UniqueName: \"kubernetes.io/projected/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-kube-api-access-rn729\") pod \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") "
Sep 11 04:11:57.594825 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.594696    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-home\") pod \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") "
Sep 11 04:11:57.594825 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.594798    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-model-cache\") pod \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") "
Sep 11 04:11:57.594961 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.594844    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tmp-dir\") pod \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") "
Sep 11 04:11:57.594961 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.594871    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-dshm\") pod \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\" (UID: \"194a5ba5-a340-46b8-b11e-5b15ab0c25cd\") "
Sep 11 04:11:57.595063 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.595026    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-home" (OuterVolumeSpecName: "home") pod "194a5ba5-a340-46b8-b11e-5b15ab0c25cd" (UID: "194a5ba5-a340-46b8-b11e-5b15ab0c25cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:57.595063 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.595041    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "194a5ba5-a340-46b8-b11e-5b15ab0c25cd" (UID: "194a5ba5-a340-46b8-b11e-5b15ab0c25cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:57.595173 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.595135    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "194a5ba5-a340-46b8-b11e-5b15ab0c25cd" (UID: "194a5ba5-a340-46b8-b11e-5b15ab0c25cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:57.595269 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.595241    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:57.595269 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.595259    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:57.595269 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.595267    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:57.596977 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.596952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-dshm" (OuterVolumeSpecName: "dshm") pod "194a5ba5-a340-46b8-b11e-5b15ab0c25cd" (UID: "194a5ba5-a340-46b8-b11e-5b15ab0c25cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:11:57.597081 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.597016    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-kube-api-access-rn729" (OuterVolumeSpecName: "kube-api-access-rn729") pod "194a5ba5-a340-46b8-b11e-5b15ab0c25cd" (UID: "194a5ba5-a340-46b8-b11e-5b15ab0c25cd"). InnerVolumeSpecName "kube-api-access-rn729". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:11:57.597081 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.597051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "194a5ba5-a340-46b8-b11e-5b15ab0c25cd" (UID: "194a5ba5-a340-46b8-b11e-5b15ab0c25cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:11:57.696450 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.696362    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:57.696450 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.696399    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:57.696450 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.696415    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn729\" (UniqueName: \"kubernetes.io/projected/194a5ba5-a340-46b8-b11e-5b15ab0c25cd-kube-api-access-rn729\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:11:57.807340 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.807301    2582 generic.go:356] "Generic (PLEG): container finished" podID="194a5ba5-a340-46b8-b11e-5b15ab0c25cd" containerID="e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5" exitCode=0
Sep 11 04:11:57.807843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.807358    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"
Sep 11 04:11:57.807843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.807391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7" event={"ID":"194a5ba5-a340-46b8-b11e-5b15ab0c25cd","Type":"ContainerDied","Data":"e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5"}
Sep 11 04:11:57.807843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.807435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7" event={"ID":"194a5ba5-a340-46b8-b11e-5b15ab0c25cd","Type":"ContainerDied","Data":"6cd3b6dbbcb4b64ed3662b55fdac833882422399a7a68067189801053fb901c6"}
Sep 11 04:11:57.807843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.807454    2582 scope.go:117] "RemoveContainer" containerID="e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5"
Sep 11 04:11:57.816786 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.816767    2582 scope.go:117] "RemoveContainer" containerID="e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5"
Sep 11 04:11:57.817095 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:11:57.817073    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5\": container with ID starting with e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5 not found: ID does not exist" containerID="e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5"
Sep 11 04:11:57.817193 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.817101    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5"} err="failed to get container status \"e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5\": rpc error: code = NotFound desc = could not find container \"e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5\": container with ID starting with e1ee1afc11bedd6042a899e7c9d0be328f76c2bc42100538e6a75af11cab89f5 not found: ID does not exist"
Sep 11 04:11:57.838016 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.837987    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"]
Sep 11 04:11:57.842413 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:57.842371    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fca08f/test-model-fca08f-kserve-55bcc6456f-dnhx7"]
Sep 11 04:11:58.274462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:11:58.274426    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="194a5ba5-a340-46b8-b11e-5b15ab0c25cd" path="/var/lib/kubelet/pods/194a5ba5-a340-46b8-b11e-5b15ab0c25cd/volumes"
Sep 11 04:12:07.810174 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:07.810137    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:12:08.101228 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.101140    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"]
Sep 11 04:12:08.101860 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.101838    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="194a5ba5-a340-46b8-b11e-5b15ab0c25cd" containerName="main"
Sep 11 04:12:08.101915 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.101866    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="194a5ba5-a340-46b8-b11e-5b15ab0c25cd" containerName="main"
Sep 11 04:12:08.101980 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.101970    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="194a5ba5-a340-46b8-b11e-5b15ab0c25cd" containerName="main"
Sep 11 04:12:08.106694 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.106668    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.109319 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.109295    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9cca98\"/\"openshift-service-ca.crt\""
Sep 11 04:12:08.109319 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.109315    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9cca98\"/\"noten-model-9cca98-kserve-self-signed-certs\""
Sep 11 04:12:08.109549 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.109295    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9cca98\"/\"kube-root-ca.crt\""
Sep 11 04:12:08.109549 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.109302    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9cca98\"/\"default-dockercfg-plng5\""
Sep 11 04:12:08.112718 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.112694    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"]
Sep 11 04:12:08.184065 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.184030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/8fb7b4ee-ab37-4780-be00-4292687c6ee6-kube-api-access-jwwmw\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.184268 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.184084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tmp-dir\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.184268 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.184151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-dshm\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.184268 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.184170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-home\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.184268 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.184222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tls-certs\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.184268 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.184253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-model-cache\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.284964 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.284932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tls-certs\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.285235 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.285211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-model-cache\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.285343 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.285329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/8fb7b4ee-ab37-4780-be00-4292687c6ee6-kube-api-access-jwwmw\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.285578 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.285525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tmp-dir\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.285794 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.285646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-model-cache\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.285893 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.285807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tmp-dir\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.285893 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.285805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-dshm\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.285893 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.285854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-home\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.286097 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.286078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-home\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.287535 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.287516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tls-certs\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.287966 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.287950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-dshm\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.293648 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.293619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/8fb7b4ee-ab37-4780-be00-4292687c6ee6-kube-api-access-jwwmw\") pod \"noten-model-9cca98-kserve-5575b5948c-hbvmk\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") " pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.417107 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.417010    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:08.694024 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.693998    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"]
Sep 11 04:12:08.696113 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:12:08.696075    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fb7b4ee_ab37_4780_be00_4292687c6ee6.slice/crio-6905a70498f0ee8e62e82f023e17a0dfe5787d167cf14e760f1a12207ffddacd WatchSource:0}: Error finding container 6905a70498f0ee8e62e82f023e17a0dfe5787d167cf14e760f1a12207ffddacd: Status 404 returned error can't find the container with id 6905a70498f0ee8e62e82f023e17a0dfe5787d167cf14e760f1a12207ffddacd
Sep 11 04:12:08.870010 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.869969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk" event={"ID":"8fb7b4ee-ab37-4780-be00-4292687c6ee6","Type":"ContainerStarted","Data":"4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848"}
Sep 11 04:12:08.870010 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:08.870022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk" event={"ID":"8fb7b4ee-ab37-4780-be00-4292687c6ee6","Type":"ContainerStarted","Data":"6905a70498f0ee8e62e82f023e17a0dfe5787d167cf14e760f1a12207ffddacd"}
Sep 11 04:12:09.957368 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:09.957312    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk" podStartSLOduration=1.957293626 podStartE2EDuration="1.957293626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:08.886436447 +0000 UTC m=+1103.150914804" watchObservedRunningTime="2026-09-11 04:12:09.957293626 +0000 UTC m=+1104.221771999"
Sep 11 04:12:09.958918 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:09.958897    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"]
Sep 11 04:12:10.860412 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:10.860376    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"]
Sep 11 04:12:10.860675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:10.860647    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl" podUID="46b508ed-7e75-4fe4-bdb1-d37320b9fed7" containerName="payload-processing" containerID="cri-o://7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7" gracePeriod=30
Sep 11 04:12:10.864333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:10.864297    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"]
Sep 11 04:12:10.884070 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:10.883877    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9" podUID="9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" containerName="payload-pre-processing" containerID="cri-o://7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521" gracePeriod=30
Sep 11 04:12:11.150274 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.150249    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:12:11.165285 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.165260    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:12:11.215781 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.215744    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-plugins-config-volume\") pod \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") "
Sep 11 04:12:11.215979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.215810    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-plugins-config-volume\") pod \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") "
Sep 11 04:12:11.215979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.215881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl5nm\" (UniqueName: \"kubernetes.io/projected/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-kube-api-access-kl5nm\") pod \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\" (UID: \"46b508ed-7e75-4fe4-bdb1-d37320b9fed7\") "
Sep 11 04:12:11.215979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.215908    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssh7p\" (UniqueName: \"kubernetes.io/projected/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-kube-api-access-ssh7p\") pod \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\" (UID: \"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4\") "
Sep 11 04:12:11.216509 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.216453    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" (UID: "9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:12:11.217720 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.216529    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46b508ed-7e75-4fe4-bdb1-d37320b9fed7" (UID: "46b508ed-7e75-4fe4-bdb1-d37320b9fed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:12:11.217720 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.216990    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:11.217720 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.217011    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:11.225262 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.225222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-kube-api-access-kl5nm" (OuterVolumeSpecName: "kube-api-access-kl5nm") pod "46b508ed-7e75-4fe4-bdb1-d37320b9fed7" (UID: "46b508ed-7e75-4fe4-bdb1-d37320b9fed7"). InnerVolumeSpecName "kube-api-access-kl5nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:11.225397 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.225283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-kube-api-access-ssh7p" (OuterVolumeSpecName: "kube-api-access-ssh7p") pod "9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" (UID: "9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4"). InnerVolumeSpecName "kube-api-access-ssh7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:11.318198 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.318155    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl5nm\" (UniqueName: \"kubernetes.io/projected/46b508ed-7e75-4fe4-bdb1-d37320b9fed7-kube-api-access-kl5nm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:11.318198 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.318193    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssh7p\" (UniqueName: \"kubernetes.io/projected/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4-kube-api-access-ssh7p\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:11.890144 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.890109    2582 generic.go:356] "Generic (PLEG): container finished" podID="46b508ed-7e75-4fe4-bdb1-d37320b9fed7" containerID="7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7" exitCode=0
Sep 11 04:12:11.890300 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.890163    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"
Sep 11 04:12:11.890300 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.890192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl" event={"ID":"46b508ed-7e75-4fe4-bdb1-d37320b9fed7","Type":"ContainerDied","Data":"7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7"}
Sep 11 04:12:11.890300 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.890254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl" event={"ID":"46b508ed-7e75-4fe4-bdb1-d37320b9fed7","Type":"ContainerDied","Data":"3f9d91882347e8decea105c6ce5940e28b1218e73a8a806ebd2bffa92d99622a"}
Sep 11 04:12:11.890300 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.890276    2582 scope.go:117] "RemoveContainer" containerID="7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7"
Sep 11 04:12:11.891882 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.891772    2582 generic.go:356] "Generic (PLEG): container finished" podID="9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" containerID="7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521" exitCode=0
Sep 11 04:12:11.891882 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.891801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9" event={"ID":"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4","Type":"ContainerDied","Data":"7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521"}
Sep 11 04:12:11.891882 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.891825    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"
Sep 11 04:12:11.891882 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.891852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9" event={"ID":"9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4","Type":"ContainerDied","Data":"18f234c811cf2d603e1dfe71b47052f3a08834f9f6bfe6c4bc9ffa3b270e8ff1"}
Sep 11 04:12:11.900044 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.900025    2582 scope.go:117] "RemoveContainer" containerID="7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7"
Sep 11 04:12:11.900327 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:12:11.900307    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7\": container with ID starting with 7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7 not found: ID does not exist" containerID="7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7"
Sep 11 04:12:11.900396 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.900339    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7"} err="failed to get container status \"7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7\": rpc error: code = NotFound desc = could not find container \"7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7\": container with ID starting with 7da3e8256a7a885e1da05f4147b9c7b7e3a04e89de3eb3c2a7e0c0da3c0149c7 not found: ID does not exist"
Sep 11 04:12:11.900396 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.900363    2582 scope.go:117] "RemoveContainer" containerID="7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521"
Sep 11 04:12:11.908576 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.908546    2582 scope.go:117] "RemoveContainer" containerID="7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521"
Sep 11 04:12:11.908864 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:12:11.908846    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521\": container with ID starting with 7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521 not found: ID does not exist" containerID="7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521"
Sep 11 04:12:11.908923 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.908874    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521"} err="failed to get container status \"7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521\": rpc error: code = NotFound desc = could not find container \"7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521\": container with ID starting with 7025d354b9fdbdb6cb714df11c50020695d1de5864d63704ebd67549ba629521 not found: ID does not exist"
Sep 11 04:12:11.913811 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.913786    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"]
Sep 11 04:12:11.917845 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.917825    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fca08f-79546d6f8c-ddzbl"]
Sep 11 04:12:11.929997 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.929965    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"]
Sep 11 04:12:11.931796 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:11.931776    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fca08f-7d8f9d6f9d-9ckl9"]
Sep 11 04:12:12.275359 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:12.275316    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b508ed-7e75-4fe4-bdb1-d37320b9fed7" path="/var/lib/kubelet/pods/46b508ed-7e75-4fe4-bdb1-d37320b9fed7/volumes"
Sep 11 04:12:12.275921 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:12.275900    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" path="/var/lib/kubelet/pods/9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4/volumes"
Sep 11 04:12:18.418096 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:18.418056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:24.176354 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.176316    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"]
Sep 11 04:12:24.176847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.176828    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" containerName="payload-pre-processing"
Sep 11 04:12:24.176888 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.176849    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" containerName="payload-pre-processing"
Sep 11 04:12:24.176888 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.176882    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b508ed-7e75-4fe4-bdb1-d37320b9fed7" containerName="payload-processing"
Sep 11 04:12:24.176888 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.176888    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b508ed-7e75-4fe4-bdb1-d37320b9fed7" containerName="payload-processing"
Sep 11 04:12:24.177043 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.176950    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1bad69-e18d-4ab0-b36a-49ec7c2b6da4" containerName="payload-pre-processing"
Sep 11 04:12:24.177043 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.176958    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b508ed-7e75-4fe4-bdb1-d37320b9fed7" containerName="payload-processing"
Sep 11 04:12:24.180065 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.180047    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.182950 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.182922    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cc6708\""
Sep 11 04:12:24.183077 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.182975    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cc6708-dockercfg-fkj92\""
Sep 11 04:12:24.191834 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.191806    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"]
Sep 11 04:12:24.248661 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.248621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94z8x\" (UniqueName: \"kubernetes.io/projected/4405beba-8ea0-47c8-a63e-2157673771b0-kube-api-access-94z8x\") pod \"payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.248858 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.248698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405beba-8ea0-47c8-a63e-2157673771b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.350083 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.350046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405beba-8ea0-47c8-a63e-2157673771b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.350275 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.350145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94z8x\" (UniqueName: \"kubernetes.io/projected/4405beba-8ea0-47c8-a63e-2157673771b0-kube-api-access-94z8x\") pod \"payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.350913 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.350892    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405beba-8ea0-47c8-a63e-2157673771b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.358586 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.358548    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94z8x\" (UniqueName: \"kubernetes.io/projected/4405beba-8ea0-47c8-a63e-2157673771b0-kube-api-access-94z8x\") pod \"payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.491846 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.491752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.769476 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.769450    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"]
Sep 11 04:12:24.772280 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:12:24.772246    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4405beba_8ea0_47c8_a63e_2157673771b0.slice/crio-c2e0a8bbd3ea02147bb311ea86797925ced9f85b845450344c3f2362c4e91b6e WatchSource:0}: Error finding container c2e0a8bbd3ea02147bb311ea86797925ced9f85b845450344c3f2362c4e91b6e: Status 404 returned error can't find the container with id c2e0a8bbd3ea02147bb311ea86797925ced9f85b845450344c3f2362c4e91b6e
Sep 11 04:12:24.965314 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.965281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6" event={"ID":"4405beba-8ea0-47c8-a63e-2157673771b0","Type":"ContainerStarted","Data":"73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060"}
Sep 11 04:12:24.965523 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.965332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6" event={"ID":"4405beba-8ea0-47c8-a63e-2157673771b0","Type":"ContainerStarted","Data":"c2e0a8bbd3ea02147bb311ea86797925ced9f85b845450344c3f2362c4e91b6e"}
Sep 11 04:12:24.965523 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.965395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:24.985162 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:24.985108    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6" podStartSLOduration=0.985092125 podStartE2EDuration="985.092125ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:24.983309423 +0000 UTC m=+1119.247787795" watchObservedRunningTime="2026-09-11 04:12:24.985092125 +0000 UTC m=+1119.249570525"
Sep 11 04:12:25.905187 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:25.905142    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk" podUID="8fb7b4ee-ab37-4780-be00-4292687c6ee6" containerName="main" containerID="cri-o://4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848" gracePeriod=45
Sep 11 04:12:26.158915 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.158856    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:26.267931 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.267893    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-home\") pod \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") "
Sep 11 04:12:26.268120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.267950    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tmp-dir\") pod \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") "
Sep 11 04:12:26.268120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.267986    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/8fb7b4ee-ab37-4780-be00-4292687c6ee6-kube-api-access-jwwmw\") pod \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") "
Sep 11 04:12:26.268120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268039    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-dshm\") pod \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") "
Sep 11 04:12:26.268291 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268224    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fb7b4ee-ab37-4780-be00-4292687c6ee6" (UID: "8fb7b4ee-ab37-4780-be00-4292687c6ee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:26.268291 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tls-certs\") pod \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") "
Sep 11 04:12:26.268291 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268273    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-home" (OuterVolumeSpecName: "home") pod "8fb7b4ee-ab37-4780-be00-4292687c6ee6" (UID: "8fb7b4ee-ab37-4780-be00-4292687c6ee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:26.268480 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268294    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-model-cache\") pod \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\" (UID: \"8fb7b4ee-ab37-4780-be00-4292687c6ee6\") "
Sep 11 04:12:26.268670 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268622    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fb7b4ee-ab37-4780-be00-4292687c6ee6" (UID: "8fb7b4ee-ab37-4780-be00-4292687c6ee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:26.268814 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268740    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:26.268814 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268759    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:26.268814 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.268773    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:26.270533 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.270483    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fb7b4ee-ab37-4780-be00-4292687c6ee6" (UID: "8fb7b4ee-ab37-4780-be00-4292687c6ee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:12:26.270774 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.270749    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-dshm" (OuterVolumeSpecName: "dshm") pod "8fb7b4ee-ab37-4780-be00-4292687c6ee6" (UID: "8fb7b4ee-ab37-4780-be00-4292687c6ee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:26.270858 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.270779    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb7b4ee-ab37-4780-be00-4292687c6ee6-kube-api-access-jwwmw" (OuterVolumeSpecName: "kube-api-access-jwwmw") pod "8fb7b4ee-ab37-4780-be00-4292687c6ee6" (UID: "8fb7b4ee-ab37-4780-be00-4292687c6ee6"). InnerVolumeSpecName "kube-api-access-jwwmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:26.369809 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.369772    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/8fb7b4ee-ab37-4780-be00-4292687c6ee6-kube-api-access-jwwmw\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:26.369809 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.369806    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb7b4ee-ab37-4780-be00-4292687c6ee6-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:26.369809 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.369817    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb7b4ee-ab37-4780-be00-4292687c6ee6-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:26.615077 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.615033    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"]
Sep 11 04:12:26.615587 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.615535    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb7b4ee-ab37-4780-be00-4292687c6ee6" containerName="main"
Sep 11 04:12:26.615587 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.615575    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb7b4ee-ab37-4780-be00-4292687c6ee6" containerName="main"
Sep 11 04:12:26.615753 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.615693    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb7b4ee-ab37-4780-be00-4292687c6ee6" containerName="main"
Sep 11 04:12:26.618961 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.618931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"
Sep 11 04:12:26.621700 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.621679    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:12:26.621801 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.621735    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:12:26.627902 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.627878    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"]
Sep 11 04:12:26.672788 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.672744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bb0d54-9fca-4355-a1ea-b00b7bfe45be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34523cde-rshbz\" (UID: \"63bb0d54-9fca-4355-a1ea-b00b7bfe45be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"
Sep 11 04:12:26.773453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.773412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bb0d54-9fca-4355-a1ea-b00b7bfe45be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34523cde-rshbz\" (UID: \"63bb0d54-9fca-4355-a1ea-b00b7bfe45be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"
Sep 11 04:12:26.774146 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.774124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bb0d54-9fca-4355-a1ea-b00b7bfe45be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34523cde-rshbz\" (UID: \"63bb0d54-9fca-4355-a1ea-b00b7bfe45be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"
Sep 11 04:12:26.929871 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.929836    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"
Sep 11 04:12:26.977361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.977330    2582 generic.go:356] "Generic (PLEG): container finished" podID="8fb7b4ee-ab37-4780-be00-4292687c6ee6" containerID="4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848" exitCode=0
Sep 11 04:12:26.977582 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.977378    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"
Sep 11 04:12:26.978209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.977381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk" event={"ID":"8fb7b4ee-ab37-4780-be00-4292687c6ee6","Type":"ContainerDied","Data":"4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848"}
Sep 11 04:12:26.978209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.977908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk" event={"ID":"8fb7b4ee-ab37-4780-be00-4292687c6ee6","Type":"ContainerDied","Data":"6905a70498f0ee8e62e82f023e17a0dfe5787d167cf14e760f1a12207ffddacd"}
Sep 11 04:12:26.978209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.977939    2582 scope.go:117] "RemoveContainer" containerID="4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848"
Sep 11 04:12:26.991738 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.991477    2582 scope.go:117] "RemoveContainer" containerID="4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848"
Sep 11 04:12:26.992455 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:12:26.992422    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848\": container with ID starting with 4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848 not found: ID does not exist" containerID="4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848"
Sep 11 04:12:26.992575 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:26.992471    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848"} err="failed to get container status \"4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848\": rpc error: code = NotFound desc = could not find container \"4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848\": container with ID starting with 4fa817796de51b6a479af16bf064bc238270fd92f203ecd448bd65a2cd02f848 not found: ID does not exist"
Sep 11 04:12:27.003011 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:27.002976    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"]
Sep 11 04:12:27.013922 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:27.013884    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-9cca98/noten-model-9cca98-kserve-5575b5948c-hbvmk"]
Sep 11 04:12:27.204462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:27.204432    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"]
Sep 11 04:12:27.207075 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:12:27.207045    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bb0d54_9fca_4355_a1ea_b00b7bfe45be.slice/crio-a75725d29c1724299e4ff333f9c4da9db4dd5fd33355843faf1a2f1ed695939c WatchSource:0}: Error finding container a75725d29c1724299e4ff333f9c4da9db4dd5fd33355843faf1a2f1ed695939c: Status 404 returned error can't find the container with id a75725d29c1724299e4ff333f9c4da9db4dd5fd33355843faf1a2f1ed695939c
Sep 11 04:12:27.994705 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:27.994669    2582 generic.go:356] "Generic (PLEG): container finished" podID="63bb0d54-9fca-4355-a1ea-b00b7bfe45be" containerID="5e7dd5be87b69be0b0625b2af5207daee577d687fa579d5a852a8ba760805bea" exitCode=0
Sep 11 04:12:27.995136 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:27.994757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz" event={"ID":"63bb0d54-9fca-4355-a1ea-b00b7bfe45be","Type":"ContainerDied","Data":"5e7dd5be87b69be0b0625b2af5207daee577d687fa579d5a852a8ba760805bea"}
Sep 11 04:12:27.995136 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:27.994839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz" event={"ID":"63bb0d54-9fca-4355-a1ea-b00b7bfe45be","Type":"ContainerStarted","Data":"a75725d29c1724299e4ff333f9c4da9db4dd5fd33355843faf1a2f1ed695939c"}
Sep 11 04:12:28.276712 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:28.276667    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb7b4ee-ab37-4780-be00-4292687c6ee6" path="/var/lib/kubelet/pods/8fb7b4ee-ab37-4780-be00-4292687c6ee6/volumes"
Sep 11 04:12:29.137726 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:29.137699    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"
Sep 11 04:12:29.302394 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:29.302307    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bb0d54-9fca-4355-a1ea-b00b7bfe45be-maas-api-service-ca\") pod \"63bb0d54-9fca-4355-a1ea-b00b7bfe45be\" (UID: \"63bb0d54-9fca-4355-a1ea-b00b7bfe45be\") "
Sep 11 04:12:29.302711 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:29.302688    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bb0d54-9fca-4355-a1ea-b00b7bfe45be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63bb0d54-9fca-4355-a1ea-b00b7bfe45be" (UID: "63bb0d54-9fca-4355-a1ea-b00b7bfe45be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:12:29.403572 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:29.403520    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bb0d54-9fca-4355-a1ea-b00b7bfe45be-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:30.009235 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:30.009197    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"
Sep 11 04:12:30.009415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:30.009231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz" event={"ID":"63bb0d54-9fca-4355-a1ea-b00b7bfe45be","Type":"ContainerDied","Data":"a75725d29c1724299e4ff333f9c4da9db4dd5fd33355843faf1a2f1ed695939c"}
Sep 11 04:12:30.009415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:30.009294    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a75725d29c1724299e4ff333f9c4da9db4dd5fd33355843faf1a2f1ed695939c"
Sep 11 04:12:35.972249 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:35.972219    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:12:37.303155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.302938    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"]
Sep 11 04:12:37.303654 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.303420    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv" podUID="197b0592-9e96-4fc8-8eae-3d38fcdda0ef" containerName="payload-processing" containerID="cri-o://60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8" gracePeriod=30
Sep 11 04:12:37.495835 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.495794    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"]
Sep 11 04:12:37.499987 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.499955    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34523cde-rshbz"]
Sep 11 04:12:37.552099 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.552065    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:12:37.687182 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.687144    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lw29\" (UniqueName: \"kubernetes.io/projected/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-kube-api-access-9lw29\") pod \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") "
Sep 11 04:12:37.687355 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.687189    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-plugins-config-volume\") pod \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\" (UID: \"197b0592-9e96-4fc8-8eae-3d38fcdda0ef\") "
Sep 11 04:12:37.687627 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.687605    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "197b0592-9e96-4fc8-8eae-3d38fcdda0ef" (UID: "197b0592-9e96-4fc8-8eae-3d38fcdda0ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:12:37.689324 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.689304    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-kube-api-access-9lw29" (OuterVolumeSpecName: "kube-api-access-9lw29") pod "197b0592-9e96-4fc8-8eae-3d38fcdda0ef" (UID: "197b0592-9e96-4fc8-8eae-3d38fcdda0ef"). InnerVolumeSpecName "kube-api-access-9lw29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:37.788076 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.788039    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lw29\" (UniqueName: \"kubernetes.io/projected/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-kube-api-access-9lw29\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:37.788076 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:37.788070    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197b0592-9e96-4fc8-8eae-3d38fcdda0ef-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:12:38.056169 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.056078    2582 generic.go:356] "Generic (PLEG): container finished" podID="197b0592-9e96-4fc8-8eae-3d38fcdda0ef" containerID="60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8" exitCode=0
Sep 11 04:12:38.056169 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.056123    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"
Sep 11 04:12:38.056169 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.056148    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv" event={"ID":"197b0592-9e96-4fc8-8eae-3d38fcdda0ef","Type":"ContainerDied","Data":"60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8"}
Sep 11 04:12:38.056407 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.056199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv" event={"ID":"197b0592-9e96-4fc8-8eae-3d38fcdda0ef","Type":"ContainerDied","Data":"b2d00dfc5ce8f00cbccbf147a2f1e343391de448f580d12c020ff5e070205d25"}
Sep 11 04:12:38.056407 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.056229    2582 scope.go:117] "RemoveContainer" containerID="60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8"
Sep 11 04:12:38.065447 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.065427    2582 scope.go:117] "RemoveContainer" containerID="60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8"
Sep 11 04:12:38.065760 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:12:38.065741    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8\": container with ID starting with 60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8 not found: ID does not exist" containerID="60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8"
Sep 11 04:12:38.065821 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.065770    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8"} err="failed to get container status \"60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8\": rpc error: code = NotFound desc = could not find container \"60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8\": container with ID starting with 60bb776a710ea43e22534ca0bdbd835a204633b1c910739139b5ff0575b3a8a8 not found: ID does not exist"
Sep 11 04:12:38.078315 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.078288    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"]
Sep 11 04:12:38.082384 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.082358    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34523cde-656f8bb88f-mr8xv"]
Sep 11 04:12:38.275531 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.275490    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="197b0592-9e96-4fc8-8eae-3d38fcdda0ef" path="/var/lib/kubelet/pods/197b0592-9e96-4fc8-8eae-3d38fcdda0ef/volumes"
Sep 11 04:12:38.275977 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:38.275956    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63bb0d54-9fca-4355-a1ea-b00b7bfe45be" path="/var/lib/kubelet/pods/63bb0d54-9fca-4355-a1ea-b00b7bfe45be/volumes"
Sep 11 04:12:53.147323 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.147290    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"]
Sep 11 04:12:53.147745 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.147722    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197b0592-9e96-4fc8-8eae-3d38fcdda0ef" containerName="payload-processing"
Sep 11 04:12:53.147745 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.147735    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="197b0592-9e96-4fc8-8eae-3d38fcdda0ef" containerName="payload-processing"
Sep 11 04:12:53.147745 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.147743    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63bb0d54-9fca-4355-a1ea-b00b7bfe45be" containerName="revoke-keys"
Sep 11 04:12:53.147850 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.147749    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="63bb0d54-9fca-4355-a1ea-b00b7bfe45be" containerName="revoke-keys"
Sep 11 04:12:53.147850 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.147833    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="63bb0d54-9fca-4355-a1ea-b00b7bfe45be" containerName="revoke-keys"
Sep 11 04:12:53.147918 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.147851    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="197b0592-9e96-4fc8-8eae-3d38fcdda0ef" containerName="payload-processing"
Sep 11 04:12:53.152550 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.152532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.155385 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.155357    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9342ed-dockercfg-jb54b\""
Sep 11 04:12:53.155547 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.155403    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9342ed\""
Sep 11 04:12:53.156491 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.156471    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"]
Sep 11 04:12:53.160133 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.160117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.164949 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.164926    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"]
Sep 11 04:12:53.185882 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.185851    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"]
Sep 11 04:12:53.230158 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.230123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg92l\" (UniqueName: \"kubernetes.io/projected/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-kube-api-access-xg92l\") pod \"payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.230303 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.230167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.230303 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.230276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.230407 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.230307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgkp\" (UniqueName: \"kubernetes.io/projected/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-kube-api-access-9xgkp\") pod \"payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.331371 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.331332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.331546 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.331388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgkp\" (UniqueName: \"kubernetes.io/projected/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-kube-api-access-9xgkp\") pod \"payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.331546 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.331474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg92l\" (UniqueName: \"kubernetes.io/projected/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-kube-api-access-xg92l\") pod \"payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.331546 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.331519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.332126 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.332106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.332223 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.332206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.340208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.340184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg92l\" (UniqueName: \"kubernetes.io/projected/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-kube-api-access-xg92l\") pod \"payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.340512 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.340478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgkp\" (UniqueName: \"kubernetes.io/projected/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-kube-api-access-9xgkp\") pod \"payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.464126 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.464090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:53.474473 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.474440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:53.752439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.752414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"]
Sep 11 04:12:53.754482 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:12:53.754449    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed87f14_37c6_4178_ba0c_aafba2fbf63c.slice/crio-b3b89ed48d3a0b697bf7890f2fcafd237d37e667732839344fd4f671277e64ea WatchSource:0}: Error finding container b3b89ed48d3a0b697bf7890f2fcafd237d37e667732839344fd4f671277e64ea: Status 404 returned error can't find the container with id b3b89ed48d3a0b697bf7890f2fcafd237d37e667732839344fd4f671277e64ea
Sep 11 04:12:53.757265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:53.757246    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"]
Sep 11 04:12:53.760029 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:12:53.760006    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975c1895_7f6a_44bb_aaa3_03bb1a7c47f1.slice/crio-5d3edfca3bf5ec3dce50ef19a3c160ffdea24bff659a9a938862ffb02f6d2adf WatchSource:0}: Error finding container 5d3edfca3bf5ec3dce50ef19a3c160ffdea24bff659a9a938862ffb02f6d2adf: Status 404 returned error can't find the container with id 5d3edfca3bf5ec3dce50ef19a3c160ffdea24bff659a9a938862ffb02f6d2adf
Sep 11 04:12:54.151414 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.151299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww" event={"ID":"3ed87f14-37c6-4178-ba0c-aafba2fbf63c","Type":"ContainerStarted","Data":"1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911"}
Sep 11 04:12:54.151414 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.151392    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww" event={"ID":"3ed87f14-37c6-4178-ba0c-aafba2fbf63c","Type":"ContainerStarted","Data":"b3b89ed48d3a0b697bf7890f2fcafd237d37e667732839344fd4f671277e64ea"}
Sep 11 04:12:54.151959 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.151427    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:12:54.152887 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.152861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz" event={"ID":"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1","Type":"ContainerStarted","Data":"d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042"}
Sep 11 04:12:54.153048 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.152904    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz" event={"ID":"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1","Type":"ContainerStarted","Data":"5d3edfca3bf5ec3dce50ef19a3c160ffdea24bff659a9a938862ffb02f6d2adf"}
Sep 11 04:12:54.153048 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.153005    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:12:54.173299 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.173244    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww" podStartSLOduration=1.173228016 podStartE2EDuration="1.173228016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:54.171452878 +0000 UTC m=+1148.435931262" watchObservedRunningTime="2026-09-11 04:12:54.173228016 +0000 UTC m=+1148.437706389"
Sep 11 04:12:54.196121 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:12:54.196067    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz" podStartSLOduration=1.196049506 podStartE2EDuration="1.196049506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:54.194686 +0000 UTC m=+1148.459164372" watchObservedRunningTime="2026-09-11 04:12:54.196049506 +0000 UTC m=+1148.460527877"
Sep 11 04:13:04.150003 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.149967    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"]
Sep 11 04:13:04.154766 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.154746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.157931 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.157912    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-772e41\""
Sep 11 04:13:04.158182 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.158166    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-772e41-dockercfg-c94sd\""
Sep 11 04:13:04.166631 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.166610    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"]
Sep 11 04:13:04.237174 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.237123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee6857-3463-47b3-a162-ab7ff586ef74-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.237361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.237241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/daee6857-3463-47b3-a162-ab7ff586ef74-kube-api-access-vdsdb\") pod \"payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.338166 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.338126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee6857-3463-47b3-a162-ab7ff586ef74-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.338388 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.338233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/daee6857-3463-47b3-a162-ab7ff586ef74-kube-api-access-vdsdb\") pod \"payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.338856 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.338833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee6857-3463-47b3-a162-ab7ff586ef74-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.349390 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.349365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/daee6857-3463-47b3-a162-ab7ff586ef74-kube-api-access-vdsdb\") pod \"payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.466715 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.466684    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:04.759980 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:04.759948    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"]
Sep 11 04:13:04.760949 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:13:04.760922    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaee6857_3463_47b3_a162_ab7ff586ef74.slice/crio-979fb6c62bfd3448b335cf02d07fb8efafc598f3e5284304f8e18b0519be6cdc WatchSource:0}: Error finding container 979fb6c62bfd3448b335cf02d07fb8efafc598f3e5284304f8e18b0519be6cdc: Status 404 returned error can't find the container with id 979fb6c62bfd3448b335cf02d07fb8efafc598f3e5284304f8e18b0519be6cdc
Sep 11 04:13:05.159680 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:05.159587    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:13:05.160024 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:05.159776    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:13:05.217047 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:05.217008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4" event={"ID":"daee6857-3463-47b3-a162-ab7ff586ef74","Type":"ContainerStarted","Data":"34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8"}
Sep 11 04:13:05.217230 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:05.217069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4" event={"ID":"daee6857-3463-47b3-a162-ab7ff586ef74","Type":"ContainerStarted","Data":"979fb6c62bfd3448b335cf02d07fb8efafc598f3e5284304f8e18b0519be6cdc"}
Sep 11 04:13:05.217230 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:05.217100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:05.235651 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:05.235588    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4" podStartSLOduration=1.235567899 podStartE2EDuration="1.235567899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:05.232426318 +0000 UTC m=+1159.496904690" watchObservedRunningTime="2026-09-11 04:13:05.235567899 +0000 UTC m=+1159.500046264"
Sep 11 04:13:16.225331 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:16.225250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:13:22.600678 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.600646    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"]
Sep 11 04:13:22.604421 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.604401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.608319 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.608298    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"kube-root-ca.crt\""
Sep 11 04:13:22.608460 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.608312    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"auth-test-model-cc6708-kserve-self-signed-certs\""
Sep 11 04:13:22.608460 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.608323    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"openshift-service-ca.crt\""
Sep 11 04:13:22.608460 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.608300    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"default-dockercfg-56nqf\""
Sep 11 04:13:22.613794 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.613772    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"]
Sep 11 04:13:22.716440 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.716405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncvt\" (UniqueName: \"kubernetes.io/projected/3f91da67-6b57-4712-acbd-dd0ca3280bb3-kube-api-access-8ncvt\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.716652 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.716466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-model-cache\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.716652 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.716542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tls-certs\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.716652 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.716634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-dshm\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.716765 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.716667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-home\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.716765 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.716735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tmp-dir\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.817830 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.817792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncvt\" (UniqueName: \"kubernetes.io/projected/3f91da67-6b57-4712-acbd-dd0ca3280bb3-kube-api-access-8ncvt\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818006 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.817861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-model-cache\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818006 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.817886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tls-certs\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818006 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.817915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-dshm\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818006 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.817942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-home\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818180 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.818006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tmp-dir\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818368 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.818345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tmp-dir\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818437 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.818367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-home\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.818437 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.818404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-model-cache\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.820354 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.820335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-dshm\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.820475 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.820456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tls-certs\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.827348 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.827328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncvt\" (UniqueName: \"kubernetes.io/projected/3f91da67-6b57-4712-acbd-dd0ca3280bb3-kube-api-access-8ncvt\") pod \"auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:22.916197 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:22.916103    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:23.205288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:23.205254    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"]
Sep 11 04:13:23.206864 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:13:23.206837    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f91da67_6b57_4712_acbd_dd0ca3280bb3.slice/crio-cc34df908c5e9e989085ff1be807d9e82c5d42fdf1cbb8e1712648aaffbab50e WatchSource:0}: Error finding container cc34df908c5e9e989085ff1be807d9e82c5d42fdf1cbb8e1712648aaffbab50e: Status 404 returned error can't find the container with id cc34df908c5e9e989085ff1be807d9e82c5d42fdf1cbb8e1712648aaffbab50e
Sep 11 04:13:23.316798 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:23.316765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h" event={"ID":"3f91da67-6b57-4712-acbd-dd0ca3280bb3","Type":"ContainerStarted","Data":"d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550"}
Sep 11 04:13:23.316968 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:23.316811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h" event={"ID":"3f91da67-6b57-4712-acbd-dd0ca3280bb3","Type":"ContainerStarted","Data":"cc34df908c5e9e989085ff1be807d9e82c5d42fdf1cbb8e1712648aaffbab50e"}
Sep 11 04:13:23.337216 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:23.337162    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h" podStartSLOduration=1.337144046 podStartE2EDuration="1.337144046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:23.335644514 +0000 UTC m=+1177.600122885" watchObservedRunningTime="2026-09-11 04:13:23.337144046 +0000 UTC m=+1177.601622737"
Sep 11 04:13:32.886228 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.886191    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"]
Sep 11 04:13:32.890984 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.890960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:32.894040 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.894015    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f8229c\""
Sep 11 04:13:32.894204 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.894187    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f8229c-dockercfg-v8nbc\""
Sep 11 04:13:32.900790 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.900768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"]
Sep 11 04:13:32.911551 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.911522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtvn\" (UniqueName: \"kubernetes.io/projected/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-kube-api-access-8dtvn\") pod \"payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:32.911725 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.911698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:32.917103 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.917074    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:32.917217 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.917111    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:32.929720 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:32.929696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:33.012814 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:33.012779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:33.012996 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:33.012923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtvn\" (UniqueName: \"kubernetes.io/projected/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-kube-api-access-8dtvn\") pod \"payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:33.013623 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:33.013604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:33.023610 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:33.023583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtvn\" (UniqueName: \"kubernetes.io/projected/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-kube-api-access-8dtvn\") pod \"payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:33.203751 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:33.203711    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:33.389747 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:33.389719    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:13:33.483264 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:33.483230    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"]
Sep 11 04:13:33.484392 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:13:33.484368    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fbfcc0_fa18_41be_a26d_85dbfe34e0a5.slice/crio-60bf3750a0ed7306e73fa460f38b91028d6f039b9ed3fb93d1d0166a6312db72 WatchSource:0}: Error finding container 60bf3750a0ed7306e73fa460f38b91028d6f039b9ed3fb93d1d0166a6312db72: Status 404 returned error can't find the container with id 60bf3750a0ed7306e73fa460f38b91028d6f039b9ed3fb93d1d0166a6312db72
Sep 11 04:13:34.383673 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:34.383633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp" event={"ID":"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5","Type":"ContainerStarted","Data":"1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d"}
Sep 11 04:13:34.384118 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:34.383686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp" event={"ID":"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5","Type":"ContainerStarted","Data":"60bf3750a0ed7306e73fa460f38b91028d6f039b9ed3fb93d1d0166a6312db72"}
Sep 11 04:13:34.384118 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:34.383810    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:34.404451 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:34.404398    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp" podStartSLOduration=2.404380861 podStartE2EDuration="2.404380861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:34.402074135 +0000 UTC m=+1188.666552507" watchObservedRunningTime="2026-09-11 04:13:34.404380861 +0000 UTC m=+1188.668859231"
Sep 11 04:13:45.391087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:45.391051    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:13:46.288383 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:46.288274    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:13:46.303100 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:13:46.290039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:14:01.002834 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.002779    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"]
Sep 11 04:14:01.014548 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.014502    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.018298 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.018257    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"]
Sep 11 04:14:01.019455 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.019436    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"kube-root-ca.crt\""
Sep 11 04:14:01.019646 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.019622    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"openshift-service-ca.crt\""
Sep 11 04:14:01.019754 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.019494    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"auth-test-model-9342ed-kserve-self-signed-certs\""
Sep 11 04:14:01.019754 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.019523    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"default-dockercfg-552lm\""
Sep 11 04:14:01.203685 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.203631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-dshm\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.203685 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.203701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-tmp-dir\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.203916 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.203742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237f745e-f6ff-4879-bb12-4f27d44fb30d-tls-certs\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.203916 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.203764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-model-cache\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.203916 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.203863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-home\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.204025 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.203919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7z6\" (UniqueName: \"kubernetes.io/projected/237f745e-f6ff-4879-bb12-4f27d44fb30d-kube-api-access-6w7z6\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.304756 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.304655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-dshm\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.304756 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.304716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-tmp-dir\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.304756 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.304741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237f745e-f6ff-4879-bb12-4f27d44fb30d-tls-certs\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.304756 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.304762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-model-cache\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.305115 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.304827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-home\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.305115 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.304883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7z6\" (UniqueName: \"kubernetes.io/projected/237f745e-f6ff-4879-bb12-4f27d44fb30d-kube-api-access-6w7z6\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.305229 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.305110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-tmp-dir\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.305229 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.305167    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-model-cache\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.305308 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.305225    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-home\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.307123 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.307100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-dshm\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.307249 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.307233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237f745e-f6ff-4879-bb12-4f27d44fb30d-tls-certs\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.313772 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.313750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7z6\" (UniqueName: \"kubernetes.io/projected/237f745e-f6ff-4879-bb12-4f27d44fb30d-kube-api-access-6w7z6\") pod \"auth-test-model-9342ed-kserve-75585444c5-h776k\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.326814 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.326786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:01.602566 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:01.602522    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"]
Sep 11 04:14:01.604472 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:14:01.604445    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod237f745e_f6ff_4879_bb12_4f27d44fb30d.slice/crio-61401b6b377250fafe6876d251c63b998aca5cdb4b06f1982c6c40e2ee8c5ed1 WatchSource:0}: Error finding container 61401b6b377250fafe6876d251c63b998aca5cdb4b06f1982c6c40e2ee8c5ed1: Status 404 returned error can't find the container with id 61401b6b377250fafe6876d251c63b998aca5cdb4b06f1982c6c40e2ee8c5ed1
Sep 11 04:14:02.542610 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:02.542540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k" event={"ID":"237f745e-f6ff-4879-bb12-4f27d44fb30d","Type":"ContainerStarted","Data":"e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67"}
Sep 11 04:14:02.543019 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:02.542628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k" event={"ID":"237f745e-f6ff-4879-bb12-4f27d44fb30d","Type":"ContainerStarted","Data":"61401b6b377250fafe6876d251c63b998aca5cdb4b06f1982c6c40e2ee8c5ed1"}
Sep 11 04:14:02.579410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:02.579353    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k" podStartSLOduration=2.579335155 podStartE2EDuration="2.579335155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:02.576722083 +0000 UTC m=+1216.841200454" watchObservedRunningTime="2026-09-11 04:14:02.579335155 +0000 UTC m=+1216.843813525"
Sep 11 04:14:10.099451 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.099411    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"]
Sep 11 04:14:10.103364 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.103343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.107309 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.107291    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-354ac6fe-dockercfg-5bjdh\""
Sep 11 04:14:10.107523 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.107510    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-354ac6fe\""
Sep 11 04:14:10.122399 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.122372    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"]
Sep 11 04:14:10.187141 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.187103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b52fab-0be3-4be3-8d63-1f224654b13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.187325 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.187165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dssl\" (UniqueName: \"kubernetes.io/projected/96b52fab-0be3-4be3-8d63-1f224654b13e-kube-api-access-5dssl\") pod \"payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.288520 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.288484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b52fab-0be3-4be3-8d63-1f224654b13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.288723 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.288544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dssl\" (UniqueName: \"kubernetes.io/projected/96b52fab-0be3-4be3-8d63-1f224654b13e-kube-api-access-5dssl\") pod \"payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.289166 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.289145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b52fab-0be3-4be3-8d63-1f224654b13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.296761 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.296734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dssl\" (UniqueName: \"kubernetes.io/projected/96b52fab-0be3-4be3-8d63-1f224654b13e-kube-api-access-5dssl\") pod \"payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.415085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.414989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:10.691165 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:10.691139    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"]
Sep 11 04:14:10.693752 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:14:10.693721    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b52fab_0be3_4be3_8d63_1f224654b13e.slice/crio-2b0a2c4a8e353dd4e9b0542c04e36d2e85ecb8bb5f02b593c8051a26241a5493 WatchSource:0}: Error finding container 2b0a2c4a8e353dd4e9b0542c04e36d2e85ecb8bb5f02b593c8051a26241a5493: Status 404 returned error can't find the container with id 2b0a2c4a8e353dd4e9b0542c04e36d2e85ecb8bb5f02b593c8051a26241a5493
Sep 11 04:14:11.327826 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.327783    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:11.328282 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.327846    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:11.340542 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.340510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:11.594333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.594233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh" event={"ID":"96b52fab-0be3-4be3-8d63-1f224654b13e","Type":"ContainerStarted","Data":"68987c901b2fd478a58c98c6b7188e4947d43481a474aac12788c5fab0a501f6"}
Sep 11 04:14:11.594333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.594298    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh" event={"ID":"96b52fab-0be3-4be3-8d63-1f224654b13e","Type":"ContainerStarted","Data":"2b0a2c4a8e353dd4e9b0542c04e36d2e85ecb8bb5f02b593c8051a26241a5493"}
Sep 11 04:14:11.594333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.594324    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:11.605795 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.605766    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:14:11.611849 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:11.611794    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh" podStartSLOduration=1.611776094 podStartE2EDuration="1.611776094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:11.610686485 +0000 UTC m=+1225.875164880" watchObservedRunningTime="2026-09-11 04:14:11.611776094 +0000 UTC m=+1225.876254467"
Sep 11 04:14:22.602154 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:22.602118    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:14:46.394656 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:14:46.394580    2582 scope.go:117] "RemoveContainer" containerID="e3a242dcffcdb8c53d7024191eb09e627bf9e1cd349b6ef5100043830dfd94c0"
Sep 11 04:15:00.160277 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.160229    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2"]
Sep 11 04:15:00.165124 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.165106    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2"
Sep 11 04:15:00.168449 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.168423    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:15:00.170678 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.170656    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"]
Sep 11 04:15:00.174074 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.174056    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"]
Sep 11 04:15:00.174190 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.174172    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"
Sep 11 04:15:00.176429 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.176410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2"
Sep 11 04:15:00.177687 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.177633    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"]
Sep 11 04:15:00.177798 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.177730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"
Sep 11 04:15:00.182266 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.182206    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2"]
Sep 11 04:15:00.182379 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.182282    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"
Sep 11 04:15:00.189988 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.189926    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"]
Sep 11 04:15:00.196467 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.196405    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"]
Sep 11 04:15:00.199409 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.199389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"]
Sep 11 04:15:00.214360 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.214337    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"
Sep 11 04:15:00.224160 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.224132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"
Sep 11 04:15:00.265288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.265060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"
Sep 11 04:15:00.469772 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.469747    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2"]
Sep 11 04:15:00.471781 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:15:00.471753    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7baa9c23_f733_4088_afff_86c1247fb59b.slice/crio-c3ac52e912e22df9b6b0f1c5d007eef2380dac9eda0038ca9dbdcd2fb63e4579 WatchSource:0}: Error finding container c3ac52e912e22df9b6b0f1c5d007eef2380dac9eda0038ca9dbdcd2fb63e4579: Status 404 returned error can't find the container with id c3ac52e912e22df9b6b0f1c5d007eef2380dac9eda0038ca9dbdcd2fb63e4579
Sep 11 04:15:00.506491 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.506463    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"]
Sep 11 04:15:00.509751 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:15:00.509719    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b744888_36e6_4b6c_823a_808edf199c49.slice/crio-2101c404704b6205d0fadfec0cbe2154b4e75a6092a1197e2bde8d08940ba382 WatchSource:0}: Error finding container 2101c404704b6205d0fadfec0cbe2154b4e75a6092a1197e2bde8d08940ba382: Status 404 returned error can't find the container with id 2101c404704b6205d0fadfec0cbe2154b4e75a6092a1197e2bde8d08940ba382
Sep 11 04:15:00.526513 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.526492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"]
Sep 11 04:15:00.528371 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:15:00.528345    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df33797_19cc_4a11_b70c_06c12ab7def9.slice/crio-f03e2e4a6c909d5af796f26a5f0b177c689dc7c3b06cbad9218d3c5dadc6e3e4 WatchSource:0}: Error finding container f03e2e4a6c909d5af796f26a5f0b177c689dc7c3b06cbad9218d3c5dadc6e3e4: Status 404 returned error can't find the container with id f03e2e4a6c909d5af796f26a5f0b177c689dc7c3b06cbad9218d3c5dadc6e3e4
Sep 11 04:15:00.576597 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.576567    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"]
Sep 11 04:15:00.577718 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:15:00.577693    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f016f52_0e52_4cb1_89e3_f5ab6cf56c3e.slice/crio-480f37298c132641d30ebed165599802b64feb29347e10861085dcb3249f7a45 WatchSource:0}: Error finding container 480f37298c132641d30ebed165599802b64feb29347e10861085dcb3249f7a45: Status 404 returned error can't find the container with id 480f37298c132641d30ebed165599802b64feb29347e10861085dcb3249f7a45
Sep 11 04:15:00.874357 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.874322    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485" event={"ID":"3df33797-19cc-4a11-b70c-06c12ab7def9","Type":"ContainerStarted","Data":"54fbefb1813050183d91090b8bcb1c8aa90dc41a9ce135aa0e20775166f830ee"}
Sep 11 04:15:00.874549 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.874378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485" event={"ID":"3df33797-19cc-4a11-b70c-06c12ab7def9","Type":"ContainerStarted","Data":"f03e2e4a6c909d5af796f26a5f0b177c689dc7c3b06cbad9218d3c5dadc6e3e4"}
Sep 11 04:15:00.875970 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.875941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq" event={"ID":"5b744888-36e6-4b6c-823a-808edf199c49","Type":"ContainerStarted","Data":"ad330c1d3138ed2ce2320042a2b304c1acb5d65d29aa42401732623f3fe2ccfa"}
Sep 11 04:15:00.876099 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.875992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq" event={"ID":"5b744888-36e6-4b6c-823a-808edf199c49","Type":"ContainerStarted","Data":"2101c404704b6205d0fadfec0cbe2154b4e75a6092a1197e2bde8d08940ba382"}
Sep 11 04:15:00.877605 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.877579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2" event={"ID":"7baa9c23-f733-4088-afff-86c1247fb59b","Type":"ContainerStarted","Data":"e8a14fe8896d21e23698a95cc79f163bcb47cd49a6c036766de6b54dc4826af7"}
Sep 11 04:15:00.877735 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.877631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2" event={"ID":"7baa9c23-f733-4088-afff-86c1247fb59b","Type":"ContainerStarted","Data":"c3ac52e912e22df9b6b0f1c5d007eef2380dac9eda0038ca9dbdcd2fb63e4579"}
Sep 11 04:15:00.879122 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.879102    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9" event={"ID":"0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e","Type":"ContainerStarted","Data":"db14fad379a631d76c694d2b604a961279397cbd82dd187a082953b7aafb8fe1"}
Sep 11 04:15:00.879242 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.879143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9" event={"ID":"0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e","Type":"ContainerStarted","Data":"480f37298c132641d30ebed165599802b64feb29347e10861085dcb3249f7a45"}
Sep 11 04:15:00.891439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.891384    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485" podStartSLOduration=0.891369483 podStartE2EDuration="891.369483ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:00.888642324 +0000 UTC m=+1275.153120695" watchObservedRunningTime="2026-09-11 04:15:00.891369483 +0000 UTC m=+1275.155847844"
Sep 11 04:15:00.905281 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.905234    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2" podStartSLOduration=0.905222088 podStartE2EDuration="905.222088ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:00.90293937 +0000 UTC m=+1275.167417743" watchObservedRunningTime="2026-09-11 04:15:00.905222088 +0000 UTC m=+1275.169700458"
Sep 11 04:15:00.918848 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.918802    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq" podStartSLOduration=0.918789956 podStartE2EDuration="918.789956ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:00.916628733 +0000 UTC m=+1275.181107105" watchObservedRunningTime="2026-09-11 04:15:00.918789956 +0000 UTC m=+1275.183268326"
Sep 11 04:15:00.932483 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:00.932438    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9" podStartSLOduration=0.932425385 podStartE2EDuration="932.425385ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:00.931074042 +0000 UTC m=+1275.195552413" watchObservedRunningTime="2026-09-11 04:15:00.932425385 +0000 UTC m=+1275.196903804"
Sep 11 04:15:01.886885 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.886845    2582 generic.go:356] "Generic (PLEG): container finished" podID="3df33797-19cc-4a11-b70c-06c12ab7def9" containerID="54fbefb1813050183d91090b8bcb1c8aa90dc41a9ce135aa0e20775166f830ee" exitCode=0
Sep 11 04:15:01.887290 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.886928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485" event={"ID":"3df33797-19cc-4a11-b70c-06c12ab7def9","Type":"ContainerDied","Data":"54fbefb1813050183d91090b8bcb1c8aa90dc41a9ce135aa0e20775166f830ee"}
Sep 11 04:15:01.888525 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.888506    2582 generic.go:356] "Generic (PLEG): container finished" podID="5b744888-36e6-4b6c-823a-808edf199c49" containerID="ad330c1d3138ed2ce2320042a2b304c1acb5d65d29aa42401732623f3fe2ccfa" exitCode=0
Sep 11 04:15:01.888624 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.888592    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq" event={"ID":"5b744888-36e6-4b6c-823a-808edf199c49","Type":"ContainerDied","Data":"ad330c1d3138ed2ce2320042a2b304c1acb5d65d29aa42401732623f3fe2ccfa"}
Sep 11 04:15:01.890010 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.889988    2582 generic.go:356] "Generic (PLEG): container finished" podID="7baa9c23-f733-4088-afff-86c1247fb59b" containerID="e8a14fe8896d21e23698a95cc79f163bcb47cd49a6c036766de6b54dc4826af7" exitCode=0
Sep 11 04:15:01.890087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.890073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2" event={"ID":"7baa9c23-f733-4088-afff-86c1247fb59b","Type":"ContainerDied","Data":"e8a14fe8896d21e23698a95cc79f163bcb47cd49a6c036766de6b54dc4826af7"}
Sep 11 04:15:01.891685 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.891661    2582 generic.go:356] "Generic (PLEG): container finished" podID="0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e" containerID="db14fad379a631d76c694d2b604a961279397cbd82dd187a082953b7aafb8fe1" exitCode=0
Sep 11 04:15:01.891770 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:01.891706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9" event={"ID":"0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e","Type":"ContainerDied","Data":"db14fad379a631d76c694d2b604a961279397cbd82dd187a082953b7aafb8fe1"}
Sep 11 04:15:03.082002 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.081972    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"
Sep 11 04:15:03.083983 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.083958    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2"
Sep 11 04:15:03.130676 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.130649    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"
Sep 11 04:15:03.134343 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.134321    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"
Sep 11 04:15:03.904766 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.904727    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"
Sep 11 04:15:03.904766 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.904761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9" event={"ID":"0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e","Type":"ContainerDied","Data":"480f37298c132641d30ebed165599802b64feb29347e10861085dcb3249f7a45"}
Sep 11 04:15:03.904987 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.904815    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="480f37298c132641d30ebed165599802b64feb29347e10861085dcb3249f7a45"
Sep 11 04:15:03.906155 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.906138    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"
Sep 11 04:15:03.906286 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.906166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485" event={"ID":"3df33797-19cc-4a11-b70c-06c12ab7def9","Type":"ContainerDied","Data":"f03e2e4a6c909d5af796f26a5f0b177c689dc7c3b06cbad9218d3c5dadc6e3e4"}
Sep 11 04:15:03.906286 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.906198    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f03e2e4a6c909d5af796f26a5f0b177c689dc7c3b06cbad9218d3c5dadc6e3e4"
Sep 11 04:15:03.907518 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.907501    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"
Sep 11 04:15:03.907640 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.907522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq" event={"ID":"5b744888-36e6-4b6c-823a-808edf199c49","Type":"ContainerDied","Data":"2101c404704b6205d0fadfec0cbe2154b4e75a6092a1197e2bde8d08940ba382"}
Sep 11 04:15:03.907640 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.907589    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2101c404704b6205d0fadfec0cbe2154b4e75a6092a1197e2bde8d08940ba382"
Sep 11 04:15:03.909045 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.909022    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2"
Sep 11 04:15:03.909045 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.909030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-rhkz2" event={"ID":"7baa9c23-f733-4088-afff-86c1247fb59b","Type":"ContainerDied","Data":"c3ac52e912e22df9b6b0f1c5d007eef2380dac9eda0038ca9dbdcd2fb63e4579"}
Sep 11 04:15:03.909163 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:03.909073    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ac52e912e22df9b6b0f1c5d007eef2380dac9eda0038ca9dbdcd2fb63e4579"
Sep 11 04:15:08.153363 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.153320    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"]
Sep 11 04:15:08.154021 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154001    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e" containerName="cleanup"
Sep 11 04:15:08.154112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154024    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e" containerName="cleanup"
Sep 11 04:15:08.154112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154056    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7baa9c23-f733-4088-afff-86c1247fb59b" containerName="cleanup"
Sep 11 04:15:08.154112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154065    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7baa9c23-f733-4088-afff-86c1247fb59b" containerName="cleanup"
Sep 11 04:15:08.154112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154082    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b744888-36e6-4b6c-823a-808edf199c49" containerName="cleanup"
Sep 11 04:15:08.154112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154091    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b744888-36e6-4b6c-823a-808edf199c49" containerName="cleanup"
Sep 11 04:15:08.154112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154107    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df33797-19cc-4a11-b70c-06c12ab7def9" containerName="cleanup"
Sep 11 04:15:08.154410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154115    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df33797-19cc-4a11-b70c-06c12ab7def9" containerName="cleanup"
Sep 11 04:15:08.154410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154219    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7baa9c23-f733-4088-afff-86c1247fb59b" containerName="cleanup"
Sep 11 04:15:08.154410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154232    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b744888-36e6-4b6c-823a-808edf199c49" containerName="cleanup"
Sep 11 04:15:08.154410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154245    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df33797-19cc-4a11-b70c-06c12ab7def9" containerName="cleanup"
Sep 11 04:15:08.154410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.154256    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e" containerName="cleanup"
Sep 11 04:15:08.157703 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.157681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"
Sep 11 04:15:08.160707 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.160685    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:15:08.160955 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.160688    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:15:08.165333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.165310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"]
Sep 11 04:15:08.189954 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.189913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1798b1-1b0a-42b4-9edb-19c4fda781d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d\" (UID: \"7c1798b1-1b0a-42b4-9edb-19c4fda781d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"
Sep 11 04:15:08.291332 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.291297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1798b1-1b0a-42b4-9edb-19c4fda781d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d\" (UID: \"7c1798b1-1b0a-42b4-9edb-19c4fda781d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"
Sep 11 04:15:08.292072 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.292051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1798b1-1b0a-42b4-9edb-19c4fda781d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d\" (UID: \"7c1798b1-1b0a-42b4-9edb-19c4fda781d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"
Sep 11 04:15:08.471646 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.471612    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"
Sep 11 04:15:08.745026 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.744999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"]
Sep 11 04:15:08.746980 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:15:08.746942    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1798b1_1b0a_42b4_9edb_19c4fda781d7.slice/crio-cec66eb5b8658c4b9ca1c20399aa56abb0abec43872add6f58e34f1b0d7187ac WatchSource:0}: Error finding container cec66eb5b8658c4b9ca1c20399aa56abb0abec43872add6f58e34f1b0d7187ac: Status 404 returned error can't find the container with id cec66eb5b8658c4b9ca1c20399aa56abb0abec43872add6f58e34f1b0d7187ac
Sep 11 04:15:08.941464 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:08.941431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d" event={"ID":"7c1798b1-1b0a-42b4-9edb-19c4fda781d7","Type":"ContainerStarted","Data":"cec66eb5b8658c4b9ca1c20399aa56abb0abec43872add6f58e34f1b0d7187ac"}
Sep 11 04:15:09.949146 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:09.949108    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c1798b1-1b0a-42b4-9edb-19c4fda781d7" containerID="c400f801c1d30d3497404a45eee7b27143de611611ea4aabffcc4372654bbe04" exitCode=0
Sep 11 04:15:09.949547 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:09.949202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d" event={"ID":"7c1798b1-1b0a-42b4-9edb-19c4fda781d7","Type":"ContainerDied","Data":"c400f801c1d30d3497404a45eee7b27143de611611ea4aabffcc4372654bbe04"}
Sep 11 04:15:11.085728 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:11.085684    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"
Sep 11 04:15:11.123250 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:11.123220    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1798b1-1b0a-42b4-9edb-19c4fda781d7-maas-api-service-ca\") pod \"7c1798b1-1b0a-42b4-9edb-19c4fda781d7\" (UID: \"7c1798b1-1b0a-42b4-9edb-19c4fda781d7\") "
Sep 11 04:15:11.123606 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:11.123582    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1798b1-1b0a-42b4-9edb-19c4fda781d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c1798b1-1b0a-42b4-9edb-19c4fda781d7" (UID: "7c1798b1-1b0a-42b4-9edb-19c4fda781d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:15:11.224246 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:11.224153    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1798b1-1b0a-42b4-9edb-19c4fda781d7-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:15:11.964415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:11.964380    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"
Sep 11 04:15:11.964415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:11.964389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d" event={"ID":"7c1798b1-1b0a-42b4-9edb-19c4fda781d7","Type":"ContainerDied","Data":"cec66eb5b8658c4b9ca1c20399aa56abb0abec43872add6f58e34f1b0d7187ac"}
Sep 11 04:15:11.964721 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:11.964445    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec66eb5b8658c4b9ca1c20399aa56abb0abec43872add6f58e34f1b0d7187ac"
Sep 11 04:15:23.867586 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:23.867519    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"]
Sep 11 04:15:23.868053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:23.867745    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh" podUID="96b52fab-0be3-4be3-8d63-1f224654b13e" containerName="payload-pre-processing" containerID="cri-o://68987c901b2fd478a58c98c6b7188e4947d43481a474aac12788c5fab0a501f6" gracePeriod=30
Sep 11 04:15:24.034363 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.034333    2582 generic.go:356] "Generic (PLEG): container finished" podID="96b52fab-0be3-4be3-8d63-1f224654b13e" containerID="68987c901b2fd478a58c98c6b7188e4947d43481a474aac12788c5fab0a501f6" exitCode=0
Sep 11 04:15:24.034530 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.034408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh" event={"ID":"96b52fab-0be3-4be3-8d63-1f224654b13e","Type":"ContainerDied","Data":"68987c901b2fd478a58c98c6b7188e4947d43481a474aac12788c5fab0a501f6"}
Sep 11 04:15:24.049547 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.049519    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"]
Sep 11 04:15:24.053198 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.053172    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-354ac6fe-45v8d"]
Sep 11 04:15:24.127919 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.127855    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:15:24.252127 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.252096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b52fab-0be3-4be3-8d63-1f224654b13e-plugins-config-volume\") pod \"96b52fab-0be3-4be3-8d63-1f224654b13e\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") "
Sep 11 04:15:24.252286 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.252236    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dssl\" (UniqueName: \"kubernetes.io/projected/96b52fab-0be3-4be3-8d63-1f224654b13e-kube-api-access-5dssl\") pod \"96b52fab-0be3-4be3-8d63-1f224654b13e\" (UID: \"96b52fab-0be3-4be3-8d63-1f224654b13e\") "
Sep 11 04:15:24.252468 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.252448    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b52fab-0be3-4be3-8d63-1f224654b13e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96b52fab-0be3-4be3-8d63-1f224654b13e" (UID: "96b52fab-0be3-4be3-8d63-1f224654b13e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:15:24.254360 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.254340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b52fab-0be3-4be3-8d63-1f224654b13e-kube-api-access-5dssl" (OuterVolumeSpecName: "kube-api-access-5dssl") pod "96b52fab-0be3-4be3-8d63-1f224654b13e" (UID: "96b52fab-0be3-4be3-8d63-1f224654b13e"). InnerVolumeSpecName "kube-api-access-5dssl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:15:24.275036 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.275005    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1798b1-1b0a-42b4-9edb-19c4fda781d7" path="/var/lib/kubelet/pods/7c1798b1-1b0a-42b4-9edb-19c4fda781d7/volumes"
Sep 11 04:15:24.353835 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.353807    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dssl\" (UniqueName: \"kubernetes.io/projected/96b52fab-0be3-4be3-8d63-1f224654b13e-kube-api-access-5dssl\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:15:24.353835 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:24.353829    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b52fab-0be3-4be3-8d63-1f224654b13e-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:15:25.041963 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:25.041930    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"
Sep 11 04:15:25.042382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:25.041927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh" event={"ID":"96b52fab-0be3-4be3-8d63-1f224654b13e","Type":"ContainerDied","Data":"2b0a2c4a8e353dd4e9b0542c04e36d2e85ecb8bb5f02b593c8051a26241a5493"}
Sep 11 04:15:25.042382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:25.042073    2582 scope.go:117] "RemoveContainer" containerID="68987c901b2fd478a58c98c6b7188e4947d43481a474aac12788c5fab0a501f6"
Sep 11 04:15:25.060487 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:25.060463    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"]
Sep 11 04:15:25.064495 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:25.064474    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-354ac6fe-7f4f64f686-nm2mh"]
Sep 11 04:15:26.275122 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:26.275089    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b52fab-0be3-4be3-8d63-1f224654b13e" path="/var/lib/kubelet/pods/96b52fab-0be3-4be3-8d63-1f224654b13e/volumes"
Sep 11 04:15:46.416577 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:15:46.416525    2582 scope.go:117] "RemoveContainer" containerID="ed7b8a47d7143ad6c654361db985e6b522a4eca054da0c8072a4754959a948a4"
Sep 11 04:16:23.794358 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:23.794326    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"]
Sep 11 04:16:25.081059 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:25.081026    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"]
Sep 11 04:16:38.813898 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:38.813835    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h" podUID="3f91da67-6b57-4712-acbd-dd0ca3280bb3" containerName="main" containerID="cri-o://d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550" gracePeriod=45
Sep 11 04:16:39.076194 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.076134    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:16:39.222220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222183    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tls-certs\") pod \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") "
Sep 11 04:16:39.222220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222222    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tmp-dir\") pod \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") "
Sep 11 04:16:39.222483 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222296    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-model-cache\") pod \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") "
Sep 11 04:16:39.222483 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ncvt\" (UniqueName: \"kubernetes.io/projected/3f91da67-6b57-4712-acbd-dd0ca3280bb3-kube-api-access-8ncvt\") pod \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") "
Sep 11 04:16:39.222483 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-home\") pod \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") "
Sep 11 04:16:39.222667 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-dshm\") pod \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\" (UID: \"3f91da67-6b57-4712-acbd-dd0ca3280bb3\") "
Sep 11 04:16:39.222667 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222538    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f91da67-6b57-4712-acbd-dd0ca3280bb3" (UID: "3f91da67-6b57-4712-acbd-dd0ca3280bb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:39.222667 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222605    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f91da67-6b57-4712-acbd-dd0ca3280bb3" (UID: "3f91da67-6b57-4712-acbd-dd0ca3280bb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:39.222836 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222723    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-home" (OuterVolumeSpecName: "home") pod "3f91da67-6b57-4712-acbd-dd0ca3280bb3" (UID: "3f91da67-6b57-4712-acbd-dd0ca3280bb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:39.222970 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222950    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:39.223045 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222974    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:39.223045 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.222987    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:39.224657 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.224632    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f91da67-6b57-4712-acbd-dd0ca3280bb3-kube-api-access-8ncvt" (OuterVolumeSpecName: "kube-api-access-8ncvt") pod "3f91da67-6b57-4712-acbd-dd0ca3280bb3" (UID: "3f91da67-6b57-4712-acbd-dd0ca3280bb3"). InnerVolumeSpecName "kube-api-access-8ncvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:16:39.224843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.224819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f91da67-6b57-4712-acbd-dd0ca3280bb3" (UID: "3f91da67-6b57-4712-acbd-dd0ca3280bb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:16:39.224942 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.224887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-dshm" (OuterVolumeSpecName: "dshm") pod "3f91da67-6b57-4712-acbd-dd0ca3280bb3" (UID: "3f91da67-6b57-4712-acbd-dd0ca3280bb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:39.324263 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.324232    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f91da67-6b57-4712-acbd-dd0ca3280bb3-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:39.324263 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.324259    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f91da67-6b57-4712-acbd-dd0ca3280bb3-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:39.324263 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.324271    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ncvt\" (UniqueName: \"kubernetes.io/projected/3f91da67-6b57-4712-acbd-dd0ca3280bb3-kube-api-access-8ncvt\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:39.470310 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.470266    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f91da67-6b57-4712-acbd-dd0ca3280bb3" containerID="d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550" exitCode=0
Sep 11 04:16:39.470511 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.470312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h" event={"ID":"3f91da67-6b57-4712-acbd-dd0ca3280bb3","Type":"ContainerDied","Data":"d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550"}
Sep 11 04:16:39.470511 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.470326    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"
Sep 11 04:16:39.470511 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.470363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h" event={"ID":"3f91da67-6b57-4712-acbd-dd0ca3280bb3","Type":"ContainerDied","Data":"cc34df908c5e9e989085ff1be807d9e82c5d42fdf1cbb8e1712648aaffbab50e"}
Sep 11 04:16:39.470511 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.470380    2582 scope.go:117] "RemoveContainer" containerID="d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550"
Sep 11 04:16:39.480407 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.480392    2582 scope.go:117] "RemoveContainer" containerID="d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550"
Sep 11 04:16:39.480729 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:16:39.480709    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550\": container with ID starting with d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550 not found: ID does not exist" containerID="d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550"
Sep 11 04:16:39.480773 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.480739    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550"} err="failed to get container status \"d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550\": rpc error: code = NotFound desc = could not find container \"d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550\": container with ID starting with d4c1ceb2e9e65c3e124dd8ae05590b30c98d559440e6a693413a20ae4bb88550 not found: ID does not exist"
Sep 11 04:16:39.492501 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.492474    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"]
Sep 11 04:16:39.499182 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:39.499162    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/auth-test-model-cc6708-kserve-56b6b5b6f5-gcv7h"]
Sep 11 04:16:40.100849 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.100809    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k" podUID="237f745e-f6ff-4879-bb12-4f27d44fb30d" containerName="main" containerID="cri-o://e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67" gracePeriod=45
Sep 11 04:16:40.276215 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.276182    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f91da67-6b57-4712-acbd-dd0ca3280bb3" path="/var/lib/kubelet/pods/3f91da67-6b57-4712-acbd-dd0ca3280bb3/volumes"
Sep 11 04:16:40.349186 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.349160    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:16:40.435668 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.435639    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-dshm\") pod \"237f745e-f6ff-4879-bb12-4f27d44fb30d\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") "
Sep 11 04:16:40.435832 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.435698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-tmp-dir\") pod \"237f745e-f6ff-4879-bb12-4f27d44fb30d\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") "
Sep 11 04:16:40.435832 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.435729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-model-cache\") pod \"237f745e-f6ff-4879-bb12-4f27d44fb30d\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") "
Sep 11 04:16:40.435832 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.435795    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-home\") pod \"237f745e-f6ff-4879-bb12-4f27d44fb30d\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") "
Sep 11 04:16:40.435832 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.435828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237f745e-f6ff-4879-bb12-4f27d44fb30d-tls-certs\") pod \"237f745e-f6ff-4879-bb12-4f27d44fb30d\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") "
Sep 11 04:16:40.436028 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.435871    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w7z6\" (UniqueName: \"kubernetes.io/projected/237f745e-f6ff-4879-bb12-4f27d44fb30d-kube-api-access-6w7z6\") pod \"237f745e-f6ff-4879-bb12-4f27d44fb30d\" (UID: \"237f745e-f6ff-4879-bb12-4f27d44fb30d\") "
Sep 11 04:16:40.436085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.436000    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "237f745e-f6ff-4879-bb12-4f27d44fb30d" (UID: "237f745e-f6ff-4879-bb12-4f27d44fb30d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:40.436085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.436015    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-model-cache" (OuterVolumeSpecName: "model-cache") pod "237f745e-f6ff-4879-bb12-4f27d44fb30d" (UID: "237f745e-f6ff-4879-bb12-4f27d44fb30d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:40.436193 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.436080    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-home" (OuterVolumeSpecName: "home") pod "237f745e-f6ff-4879-bb12-4f27d44fb30d" (UID: "237f745e-f6ff-4879-bb12-4f27d44fb30d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:40.436265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.436246    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:40.436302 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.436269    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:40.436302 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.436284    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:40.437921 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.437891    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-dshm" (OuterVolumeSpecName: "dshm") pod "237f745e-f6ff-4879-bb12-4f27d44fb30d" (UID: "237f745e-f6ff-4879-bb12-4f27d44fb30d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:40.438040 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.437969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/237f745e-f6ff-4879-bb12-4f27d44fb30d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "237f745e-f6ff-4879-bb12-4f27d44fb30d" (UID: "237f745e-f6ff-4879-bb12-4f27d44fb30d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:16:40.438089 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.438048    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237f745e-f6ff-4879-bb12-4f27d44fb30d-kube-api-access-6w7z6" (OuterVolumeSpecName: "kube-api-access-6w7z6") pod "237f745e-f6ff-4879-bb12-4f27d44fb30d" (UID: "237f745e-f6ff-4879-bb12-4f27d44fb30d"). InnerVolumeSpecName "kube-api-access-6w7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:16:40.477709 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.477674    2582 generic.go:356] "Generic (PLEG): container finished" podID="237f745e-f6ff-4879-bb12-4f27d44fb30d" containerID="e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67" exitCode=0
Sep 11 04:16:40.477887 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.477715    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"
Sep 11 04:16:40.477887 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.477763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k" event={"ID":"237f745e-f6ff-4879-bb12-4f27d44fb30d","Type":"ContainerDied","Data":"e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67"}
Sep 11 04:16:40.477887 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.477810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k" event={"ID":"237f745e-f6ff-4879-bb12-4f27d44fb30d","Type":"ContainerDied","Data":"61401b6b377250fafe6876d251c63b998aca5cdb4b06f1982c6c40e2ee8c5ed1"}
Sep 11 04:16:40.477887 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.477837    2582 scope.go:117] "RemoveContainer" containerID="e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67"
Sep 11 04:16:40.487842 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.487821    2582 scope.go:117] "RemoveContainer" containerID="e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67"
Sep 11 04:16:40.488109 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:16:40.488090    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67\": container with ID starting with e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67 not found: ID does not exist" containerID="e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67"
Sep 11 04:16:40.488183 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.488116    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67"} err="failed to get container status \"e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67\": rpc error: code = NotFound desc = could not find container \"e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67\": container with ID starting with e0924de1ee0aa1726ce48fdd12fc12fd1a00f65d44de7cf48ffe21f0fcf7da67 not found: ID does not exist"
Sep 11 04:16:40.502640 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.502614    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"]
Sep 11 04:16:40.508797 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.508777    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/auth-test-model-9342ed-kserve-75585444c5-h776k"]
Sep 11 04:16:40.536895 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.536863    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/237f745e-f6ff-4879-bb12-4f27d44fb30d-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:40.536895 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.536892    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w7z6\" (UniqueName: \"kubernetes.io/projected/237f745e-f6ff-4879-bb12-4f27d44fb30d-kube-api-access-6w7z6\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:40.537081 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:40.536902    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/237f745e-f6ff-4879-bb12-4f27d44fb30d-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:16:42.275649 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:42.275612    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="237f745e-f6ff-4879-bb12-4f27d44fb30d" path="/var/lib/kubelet/pods/237f745e-f6ff-4879-bb12-4f27d44fb30d/volumes"
Sep 11 04:16:46.442985 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:46.442956    2582 scope.go:117] "RemoveContainer" containerID="b3de039ee033a7a636946c55959fc2ce83136ce02e8fab24d2e0515927b40624"
Sep 11 04:16:46.452162 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:46.452139    2582 scope.go:117] "RemoveContainer" containerID="7348d3129d19110be677e2464f53910941b4cf9be1e0d7a9aec976dafdad0055"
Sep 11 04:16:49.004571 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.004519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"]
Sep 11 04:16:49.005029 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005011    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b52fab-0be3-4be3-8d63-1f224654b13e" containerName="payload-pre-processing"
Sep 11 04:16:49.005085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005031    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b52fab-0be3-4be3-8d63-1f224654b13e" containerName="payload-pre-processing"
Sep 11 04:16:49.005085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005042    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f91da67-6b57-4712-acbd-dd0ca3280bb3" containerName="main"
Sep 11 04:16:49.005085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005048    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f91da67-6b57-4712-acbd-dd0ca3280bb3" containerName="main"
Sep 11 04:16:49.005085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005059    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="237f745e-f6ff-4879-bb12-4f27d44fb30d" containerName="main"
Sep 11 04:16:49.005085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005064    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="237f745e-f6ff-4879-bb12-4f27d44fb30d" containerName="main"
Sep 11 04:16:49.005085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005077    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1798b1-1b0a-42b4-9edb-19c4fda781d7" containerName="revoke-keys"
Sep 11 04:16:49.005085 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005082    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1798b1-1b0a-42b4-9edb-19c4fda781d7" containerName="revoke-keys"
Sep 11 04:16:49.005325 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005168    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b52fab-0be3-4be3-8d63-1f224654b13e" containerName="payload-pre-processing"
Sep 11 04:16:49.005325 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005178    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="237f745e-f6ff-4879-bb12-4f27d44fb30d" containerName="main"
Sep 11 04:16:49.005325 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005184    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1798b1-1b0a-42b4-9edb-19c4fda781d7" containerName="revoke-keys"
Sep 11 04:16:49.005325 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.005192    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f91da67-6b57-4712-acbd-dd0ca3280bb3" containerName="main"
Sep 11 04:16:49.009649 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.009626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.013464 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.013441    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"default-dockercfg-56nqf\""
Sep 11 04:16:49.013596 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.013461    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"kube-root-ca.crt\""
Sep 11 04:16:49.013596 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.013442    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"sub-test-model-cc6708-kserve-self-signed-certs\""
Sep 11 04:16:49.013596 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.013441    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"openshift-service-ca.crt\""
Sep 11 04:16:49.018012 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.017990    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"]
Sep 11 04:16:49.120913 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.120876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-dshm\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.120913 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.120916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-model-cache\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.121138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.120934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-home\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.121138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.121022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwq2x\" (UniqueName: \"kubernetes.io/projected/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-kube-api-access-pwq2x\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.121138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.121100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tls-certs\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.121248 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.121146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tmp-dir\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.221855 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.221821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tls-certs\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222058 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.221860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tmp-dir\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222058 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.221930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-dshm\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222058 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.221953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-model-cache\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222058 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.222014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-home\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222274 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.222075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwq2x\" (UniqueName: \"kubernetes.io/projected/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-kube-api-access-pwq2x\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222384 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.222358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-model-cache\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.222407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tmp-dir\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.222566 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.222536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-home\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.224268 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.224245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-dshm\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.224414 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.224395    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tls-certs\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.230803 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.230774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwq2x\" (UniqueName: \"kubernetes.io/projected/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-kube-api-access-pwq2x\") pod \"sub-test-model-cc6708-kserve-6fbcb99989-795mp\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.320619 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.320500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:49.594193 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:49.594158    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"]
Sep 11 04:16:49.595785 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:16:49.595749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6145a7ac_12f2_4bc6_abad_cfe5143f1d9e.slice/crio-8a325d824458a933acec001ad352d358d85769e36d632a39c4af6e81b44a0ce9 WatchSource:0}: Error finding container 8a325d824458a933acec001ad352d358d85769e36d632a39c4af6e81b44a0ce9: Status 404 returned error can't find the container with id 8a325d824458a933acec001ad352d358d85769e36d632a39c4af6e81b44a0ce9
Sep 11 04:16:50.545003 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:50.544962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp" event={"ID":"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e","Type":"ContainerStarted","Data":"32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73"}
Sep 11 04:16:50.545404 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:50.545016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp" event={"ID":"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e","Type":"ContainerStarted","Data":"8a325d824458a933acec001ad352d358d85769e36d632a39c4af6e81b44a0ce9"}
Sep 11 04:16:50.563414 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:50.563358    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp" podStartSLOduration=2.563339341 podStartE2EDuration="2.563339341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:16:50.561982708 +0000 UTC m=+1384.826461078" watchObservedRunningTime="2026-09-11 04:16:50.563339341 +0000 UTC m=+1384.827817712"
Sep 11 04:16:59.321102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:59.321062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:59.321102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:59.321109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:59.333805 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:59.333775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:16:59.604960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:16:59.604870    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:17:31.428149 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.428113    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"]
Sep 11 04:17:31.432199 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.432179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.436322 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.436291    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bf30f8-dockercfg-5c4p9\""
Sep 11 04:17:31.436834 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.436811    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bf30f8\""
Sep 11 04:17:31.445138 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.445114    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"]
Sep 11 04:17:31.479875 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.479839    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"]
Sep 11 04:17:31.483738 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.483721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:31.494657 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.494627    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"]
Sep 11 04:17:31.553033 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.553000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwf2\" (UniqueName: \"kubernetes.io/projected/5e8a1711-e42e-4d48-90d8-b96d9ff50160-kube-api-access-vrwf2\") pod \"payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.553033 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.553038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8a1711-e42e-4d48-90d8-b96d9ff50160-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.553227 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.553148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgc7\" (UniqueName: \"kubernetes.io/projected/4fbdf18b-7100-43f5-abf7-3eedf013b05c-kube-api-access-lqgc7\") pod \"payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:31.553227 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.553179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdf18b-7100-43f5-abf7-3eedf013b05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:31.654093 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.654053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgc7\" (UniqueName: \"kubernetes.io/projected/4fbdf18b-7100-43f5-abf7-3eedf013b05c-kube-api-access-lqgc7\") pod \"payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:31.654093 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.654098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdf18b-7100-43f5-abf7-3eedf013b05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:31.654304 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.654155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwf2\" (UniqueName: \"kubernetes.io/projected/5e8a1711-e42e-4d48-90d8-b96d9ff50160-kube-api-access-vrwf2\") pod \"payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.654304 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.654175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8a1711-e42e-4d48-90d8-b96d9ff50160-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.654925 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.654903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdf18b-7100-43f5-abf7-3eedf013b05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:31.654976 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.654895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8a1711-e42e-4d48-90d8-b96d9ff50160-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.662241 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.662220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwf2\" (UniqueName: \"kubernetes.io/projected/5e8a1711-e42e-4d48-90d8-b96d9ff50160-kube-api-access-vrwf2\") pod \"payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") " pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.662379 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.662358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgc7\" (UniqueName: \"kubernetes.io/projected/4fbdf18b-7100-43f5-abf7-3eedf013b05c-kube-api-access-lqgc7\") pod \"payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:31.750975 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.750891    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:31.795111 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:31.795073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:32.027208 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.027179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"]
Sep 11 04:17:32.028873 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:17:32.028845    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8a1711_e42e_4d48_90d8_b96d9ff50160.slice/crio-ac29ed62945cd71d333a7248098390cda8a9f90543568eb15055bf829a1b1596 WatchSource:0}: Error finding container ac29ed62945cd71d333a7248098390cda8a9f90543568eb15055bf829a1b1596: Status 404 returned error can't find the container with id ac29ed62945cd71d333a7248098390cda8a9f90543568eb15055bf829a1b1596
Sep 11 04:17:32.082701 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.082676    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"]
Sep 11 04:17:32.085307 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:17:32.085277    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbdf18b_7100_43f5_abf7_3eedf013b05c.slice/crio-02512c0ffa184c10d766ac5f4a66999f0618e287db3f305cfccd4a981a51f604 WatchSource:0}: Error finding container 02512c0ffa184c10d766ac5f4a66999f0618e287db3f305cfccd4a981a51f604: Status 404 returned error can't find the container with id 02512c0ffa184c10d766ac5f4a66999f0618e287db3f305cfccd4a981a51f604
Sep 11 04:17:32.782843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.782803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl" event={"ID":"4fbdf18b-7100-43f5-abf7-3eedf013b05c","Type":"ContainerStarted","Data":"63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b"}
Sep 11 04:17:32.783331 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.782860    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl" event={"ID":"4fbdf18b-7100-43f5-abf7-3eedf013b05c","Type":"ContainerStarted","Data":"02512c0ffa184c10d766ac5f4a66999f0618e287db3f305cfccd4a981a51f604"}
Sep 11 04:17:32.783331 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.782998    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:32.784537 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.784510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg" event={"ID":"5e8a1711-e42e-4d48-90d8-b96d9ff50160","Type":"ContainerStarted","Data":"35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd"}
Sep 11 04:17:32.784675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.784578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg" event={"ID":"5e8a1711-e42e-4d48-90d8-b96d9ff50160","Type":"ContainerStarted","Data":"ac29ed62945cd71d333a7248098390cda8a9f90543568eb15055bf829a1b1596"}
Sep 11 04:17:32.784675 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.784600    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:32.803109 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.803063    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl" podStartSLOduration=1.803048842 podStartE2EDuration="1.803048842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:17:32.800806896 +0000 UTC m=+1427.065285270" watchObservedRunningTime="2026-09-11 04:17:32.803048842 +0000 UTC m=+1427.067527253"
Sep 11 04:17:32.821400 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:32.821340    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg" podStartSLOduration=1.821320557 podStartE2EDuration="1.821320557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:17:32.817610354 +0000 UTC m=+1427.082088727" watchObservedRunningTime="2026-09-11 04:17:32.821320557 +0000 UTC m=+1427.085798929"
Sep 11 04:17:43.792783 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:43.792699    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:17:43.793255 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:43.792803    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:17:46.487533 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:17:46.487497    2582 scope.go:117] "RemoveContainer" containerID="819f4270ce5c45de2d98718d843ee750aed044a9544fcacea6a177464a715b5e"
Sep 11 04:18:00.460377 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.460339    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"]
Sep 11 04:18:00.465544 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.465522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.468256 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.468233    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-599baf-dockercfg-n5x7j\""
Sep 11 04:18:00.468521 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.468505    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-599baf\""
Sep 11 04:18:00.479415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.479382    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"]
Sep 11 04:18:00.482492 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.482454    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"]
Sep 11 04:18:00.486431 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.486411    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:00.499988 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.499961    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"]
Sep 11 04:18:00.554691 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.554657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm56w\" (UniqueName: \"kubernetes.io/projected/b513424e-0955-4d55-9331-6e7ef6d3a8ea-kube-api-access-lm56w\") pod \"payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.554884 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.554737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxdl\" (UniqueName: \"kubernetes.io/projected/3e637b70-180e-4ee4-b942-20a26cffb917-kube-api-access-htxdl\") pod \"payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") " pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:00.554884 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.554798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b513424e-0955-4d55-9331-6e7ef6d3a8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.554884 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.554855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e637b70-180e-4ee4-b942-20a26cffb917-plugins-config-volume\") pod \"payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") " pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:00.656285 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.656239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm56w\" (UniqueName: \"kubernetes.io/projected/b513424e-0955-4d55-9331-6e7ef6d3a8ea-kube-api-access-lm56w\") pod \"payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.656498 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.656308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxdl\" (UniqueName: \"kubernetes.io/projected/3e637b70-180e-4ee4-b942-20a26cffb917-kube-api-access-htxdl\") pod \"payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") " pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:00.656498 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.656336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b513424e-0955-4d55-9331-6e7ef6d3a8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.656498 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.656371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e637b70-180e-4ee4-b942-20a26cffb917-plugins-config-volume\") pod \"payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") " pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:00.657136 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.657115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e637b70-180e-4ee4-b942-20a26cffb917-plugins-config-volume\") pod \"payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") " pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:00.657178 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.657126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b513424e-0955-4d55-9331-6e7ef6d3a8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.664689 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.664655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxdl\" (UniqueName: \"kubernetes.io/projected/3e637b70-180e-4ee4-b942-20a26cffb917-kube-api-access-htxdl\") pod \"payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") " pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:00.664844 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.664820    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm56w\" (UniqueName: \"kubernetes.io/projected/b513424e-0955-4d55-9331-6e7ef6d3a8ea-kube-api-access-lm56w\") pod \"payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.776077 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.775984    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:00.798105 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:00.798074    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:01.058965 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.058935    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"]
Sep 11 04:18:01.060857 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:18:01.060828    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb513424e_0955_4d55_9331_6e7ef6d3a8ea.slice/crio-c9af06c941ee88da82813f32747e04fbcbc49e434730fc474118f6074c5c9cf3 WatchSource:0}: Error finding container c9af06c941ee88da82813f32747e04fbcbc49e434730fc474118f6074c5c9cf3: Status 404 returned error can't find the container with id c9af06c941ee88da82813f32747e04fbcbc49e434730fc474118f6074c5c9cf3
Sep 11 04:18:01.092189 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.092164    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"]
Sep 11 04:18:01.097726 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:18:01.097697    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e637b70_180e_4ee4_b942_20a26cffb917.slice/crio-b213f0b2fddc5cc6593518d8e6485b5e8adf80bec9bd1429955739da869987c1 WatchSource:0}: Error finding container b213f0b2fddc5cc6593518d8e6485b5e8adf80bec9bd1429955739da869987c1: Status 404 returned error can't find the container with id b213f0b2fddc5cc6593518d8e6485b5e8adf80bec9bd1429955739da869987c1
Sep 11 04:18:01.967909 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.967869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8" event={"ID":"b513424e-0955-4d55-9331-6e7ef6d3a8ea","Type":"ContainerStarted","Data":"ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b"}
Sep 11 04:18:01.968408 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.967943    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:01.968408 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.967964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8" event={"ID":"b513424e-0955-4d55-9331-6e7ef6d3a8ea","Type":"ContainerStarted","Data":"c9af06c941ee88da82813f32747e04fbcbc49e434730fc474118f6074c5c9cf3"}
Sep 11 04:18:01.969510 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.969478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4" event={"ID":"3e637b70-180e-4ee4-b942-20a26cffb917","Type":"ContainerStarted","Data":"759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4"}
Sep 11 04:18:01.969671 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.969534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4" event={"ID":"3e637b70-180e-4ee4-b942-20a26cffb917","Type":"ContainerStarted","Data":"b213f0b2fddc5cc6593518d8e6485b5e8adf80bec9bd1429955739da869987c1"}
Sep 11 04:18:01.969671 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.969579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:01.985820 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:01.985769    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8" podStartSLOduration=1.985752659 podStartE2EDuration="1.985752659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:18:01.983780054 +0000 UTC m=+1456.248258437" watchObservedRunningTime="2026-09-11 04:18:01.985752659 +0000 UTC m=+1456.250231030"
Sep 11 04:18:02.001142 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:02.001081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4" podStartSLOduration=2.00106349 podStartE2EDuration="2.00106349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:18:02.000530196 +0000 UTC m=+1456.265008582" watchObservedRunningTime="2026-09-11 04:18:02.00106349 +0000 UTC m=+1456.265541861"
Sep 11 04:18:12.976747 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:12.976715    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:18:12.977211 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:12.976892    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:18:14.778612 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:14.778577    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"]
Sep 11 04:18:26.771462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:26.771421    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"]
Sep 11 04:18:26.775412 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:26.775392    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"
Sep 11 04:18:26.778200 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:26.778182    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:18:26.778280 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:26.778234    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:18:26.782477 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:26.782448    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"]
Sep 11 04:18:26.926081 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:26.926024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth\" (UID: \"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"
Sep 11 04:18:27.027459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:27.027355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth\" (UID: \"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"
Sep 11 04:18:27.028035 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:27.028012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth\" (UID: \"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"
Sep 11 04:18:27.089216 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:27.089178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"
Sep 11 04:18:27.371357 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:27.371323    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"]
Sep 11 04:18:27.373409 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:18:27.373378    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b1c8e_3ba4_4d62_94a2_1dbb61b6547c.slice/crio-ed0a4bf647cfed0a049352100bc25ceba30deb180051f83832fbb5c91cc4db69 WatchSource:0}: Error finding container ed0a4bf647cfed0a049352100bc25ceba30deb180051f83832fbb5c91cc4db69: Status 404 returned error can't find the container with id ed0a4bf647cfed0a049352100bc25ceba30deb180051f83832fbb5c91cc4db69
Sep 11 04:18:28.131626 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:28.131589    2582 generic.go:356] "Generic (PLEG): container finished" podID="d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c" containerID="86e475dfab78922437c903cbde2adca76ae0c404f3004601731afb5cc56dccd6" exitCode=0
Sep 11 04:18:28.132053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:28.131626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth" event={"ID":"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c","Type":"ContainerDied","Data":"86e475dfab78922437c903cbde2adca76ae0c404f3004601731afb5cc56dccd6"}
Sep 11 04:18:28.132053 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:28.131726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth" event={"ID":"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c","Type":"ContainerStarted","Data":"ed0a4bf647cfed0a049352100bc25ceba30deb180051f83832fbb5c91cc4db69"}
Sep 11 04:18:29.279331 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:29.279303    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"
Sep 11 04:18:29.347910 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:29.347878    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c-maas-api-service-ca\") pod \"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c\" (UID: \"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c\") "
Sep 11 04:18:29.348244 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:29.348218    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c" (UID: "d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:18:29.449471 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:29.449436    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:29.798660 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:29.798546    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp" podUID="6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" containerName="main" containerID="cri-o://32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73" gracePeriod=45
Sep 11 04:18:30.055288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.055220    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:18:30.145301 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.145267    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"
Sep 11 04:18:30.145480 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.145308    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth" event={"ID":"d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c","Type":"ContainerDied","Data":"ed0a4bf647cfed0a049352100bc25ceba30deb180051f83832fbb5c91cc4db69"}
Sep 11 04:18:30.145480 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.145365    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed0a4bf647cfed0a049352100bc25ceba30deb180051f83832fbb5c91cc4db69"
Sep 11 04:18:30.146839 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.146810    2582 generic.go:356] "Generic (PLEG): container finished" podID="6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" containerID="32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73" exitCode=0
Sep 11 04:18:30.146949 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.146840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp" event={"ID":"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e","Type":"ContainerDied","Data":"32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73"}
Sep 11 04:18:30.146949 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.146883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp" event={"ID":"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e","Type":"ContainerDied","Data":"8a325d824458a933acec001ad352d358d85769e36d632a39c4af6e81b44a0ce9"}
Sep 11 04:18:30.146949 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.146902    2582 scope.go:117] "RemoveContainer" containerID="32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73"
Sep 11 04:18:30.146949 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.146850    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"
Sep 11 04:18:30.156373 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156340    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tmp-dir\") pod \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") "
Sep 11 04:18:30.156439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-dshm\") pod \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") "
Sep 11 04:18:30.156439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwq2x\" (UniqueName: \"kubernetes.io/projected/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-kube-api-access-pwq2x\") pod \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") "
Sep 11 04:18:30.156548 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156410    2582 scope.go:117] "RemoveContainer" containerID="32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73"
Sep 11 04:18:30.156548 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tls-certs\") pod \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") "
Sep 11 04:18:30.156695 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156598    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-model-cache\") pod \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") "
Sep 11 04:18:30.156695 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156666    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" (UID: "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:30.156797 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156722    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-home\") pod \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\" (UID: \"6145a7ac-12f2-4bc6-abad-cfe5143f1d9e\") "
Sep 11 04:18:30.156856 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:18:30.156827    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73\": container with ID starting with 32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73 not found: ID does not exist" containerID="32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73"
Sep 11 04:18:30.156914 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.156868    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73"} err="failed to get container status \"32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73\": rpc error: code = NotFound desc = could not find container \"32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73\": container with ID starting with 32d027505cb7ac86c9148d739c80cc0e0bc7d8e8148e9b20580fc7b53f7f8a73 not found: ID does not exist"
Sep 11 04:18:30.157120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.157098    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" (UID: "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:30.157120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.157113    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:30.157314 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.157298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-home" (OuterVolumeSpecName: "home") pod "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" (UID: "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:30.158874 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.158852    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-kube-api-access-pwq2x" (OuterVolumeSpecName: "kube-api-access-pwq2x") pod "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" (UID: "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e"). InnerVolumeSpecName "kube-api-access-pwq2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:18:30.159017 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.158994    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" (UID: "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:18:30.159060 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.159004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-dshm" (OuterVolumeSpecName: "dshm") pod "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" (UID: "6145a7ac-12f2-4bc6-abad-cfe5143f1d9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:30.258492 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.258449    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:30.258492 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.258480    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwq2x\" (UniqueName: \"kubernetes.io/projected/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-kube-api-access-pwq2x\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:30.258492 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.258490    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:30.258492 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.258499    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:30.258875 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.258506    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:30.464343 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.464307    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"]
Sep 11 04:18:30.471221 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:30.471188    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/sub-test-model-cc6708-kserve-6fbcb99989-795mp"]
Sep 11 04:18:32.275604 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:32.275551    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" path="/var/lib/kubelet/pods/6145a7ac-12f2-4bc6-abad-cfe5143f1d9e/volumes"
Sep 11 04:18:42.284192 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.284155    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"]
Sep 11 04:18:42.284651 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.284432    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg" podUID="5e8a1711-e42e-4d48-90d8-b96d9ff50160" containerName="payload-processing" containerID="cri-o://35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd" gracePeriod=30
Sep 11 04:18:42.298484 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.298452    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"]
Sep 11 04:18:42.298931 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.298899    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl" podUID="4fbdf18b-7100-43f5-abf7-3eedf013b05c" containerName="payload-pre-processing" containerID="cri-o://63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b" gracePeriod=30
Sep 11 04:18:42.470144 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.470067    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"]
Sep 11 04:18:42.473449 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.473392    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bf30f8-d7hth"]
Sep 11 04:18:42.550495 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.550430    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:18:42.582042 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.582010    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:18:42.693965 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.693929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqgc7\" (UniqueName: \"kubernetes.io/projected/4fbdf18b-7100-43f5-abf7-3eedf013b05c-kube-api-access-lqgc7\") pod \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") "
Sep 11 04:18:42.694142 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.693996    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdf18b-7100-43f5-abf7-3eedf013b05c-plugins-config-volume\") pod \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\" (UID: \"4fbdf18b-7100-43f5-abf7-3eedf013b05c\") "
Sep 11 04:18:42.694142 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.694029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8a1711-e42e-4d48-90d8-b96d9ff50160-plugins-config-volume\") pod \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") "
Sep 11 04:18:42.694259 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.694145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrwf2\" (UniqueName: \"kubernetes.io/projected/5e8a1711-e42e-4d48-90d8-b96d9ff50160-kube-api-access-vrwf2\") pod \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\" (UID: \"5e8a1711-e42e-4d48-90d8-b96d9ff50160\") "
Sep 11 04:18:42.694487 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.694450    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbdf18b-7100-43f5-abf7-3eedf013b05c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fbdf18b-7100-43f5-abf7-3eedf013b05c" (UID: "4fbdf18b-7100-43f5-abf7-3eedf013b05c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:18:42.694645 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.694511    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8a1711-e42e-4d48-90d8-b96d9ff50160-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e8a1711-e42e-4d48-90d8-b96d9ff50160" (UID: "5e8a1711-e42e-4d48-90d8-b96d9ff50160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:18:42.694645 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.694528    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbdf18b-7100-43f5-abf7-3eedf013b05c-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:42.696465 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.696439    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbdf18b-7100-43f5-abf7-3eedf013b05c-kube-api-access-lqgc7" (OuterVolumeSpecName: "kube-api-access-lqgc7") pod "4fbdf18b-7100-43f5-abf7-3eedf013b05c" (UID: "4fbdf18b-7100-43f5-abf7-3eedf013b05c"). InnerVolumeSpecName "kube-api-access-lqgc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:18:42.696581 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.696468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8a1711-e42e-4d48-90d8-b96d9ff50160-kube-api-access-vrwf2" (OuterVolumeSpecName: "kube-api-access-vrwf2") pod "5e8a1711-e42e-4d48-90d8-b96d9ff50160" (UID: "5e8a1711-e42e-4d48-90d8-b96d9ff50160"). InnerVolumeSpecName "kube-api-access-vrwf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:18:42.796007 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.795968    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrwf2\" (UniqueName: \"kubernetes.io/projected/5e8a1711-e42e-4d48-90d8-b96d9ff50160-kube-api-access-vrwf2\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:42.796007 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.796003    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgc7\" (UniqueName: \"kubernetes.io/projected/4fbdf18b-7100-43f5-abf7-3eedf013b05c-kube-api-access-lqgc7\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:42.796007 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:42.796012    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8a1711-e42e-4d48-90d8-b96d9ff50160-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:43.226927 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.226889    2582 generic.go:356] "Generic (PLEG): container finished" podID="4fbdf18b-7100-43f5-abf7-3eedf013b05c" containerID="63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b" exitCode=0
Sep 11 04:18:43.226927 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.226929    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"
Sep 11 04:18:43.227199 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.226977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl" event={"ID":"4fbdf18b-7100-43f5-abf7-3eedf013b05c","Type":"ContainerDied","Data":"63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b"}
Sep 11 04:18:43.227199 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.227039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl" event={"ID":"4fbdf18b-7100-43f5-abf7-3eedf013b05c","Type":"ContainerDied","Data":"02512c0ffa184c10d766ac5f4a66999f0618e287db3f305cfccd4a981a51f604"}
Sep 11 04:18:43.227199 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.227071    2582 scope.go:117] "RemoveContainer" containerID="63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b"
Sep 11 04:18:43.228626 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.228599    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e8a1711-e42e-4d48-90d8-b96d9ff50160" containerID="35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd" exitCode=0
Sep 11 04:18:43.228744 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.228647    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"
Sep 11 04:18:43.228744 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.228646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg" event={"ID":"5e8a1711-e42e-4d48-90d8-b96d9ff50160","Type":"ContainerDied","Data":"35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd"}
Sep 11 04:18:43.228744 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.228698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg" event={"ID":"5e8a1711-e42e-4d48-90d8-b96d9ff50160","Type":"ContainerDied","Data":"ac29ed62945cd71d333a7248098390cda8a9f90543568eb15055bf829a1b1596"}
Sep 11 04:18:43.238344 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.238324    2582 scope.go:117] "RemoveContainer" containerID="63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b"
Sep 11 04:18:43.238715 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:18:43.238691    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b\": container with ID starting with 63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b not found: ID does not exist" containerID="63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b"
Sep 11 04:18:43.238806 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.238724    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b"} err="failed to get container status \"63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b\": rpc error: code = NotFound desc = could not find container \"63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b\": container with ID starting with 63f5a1addaef9aeba5562b3a3fb241d82762fd70dea073c7a7c4a7f156277f1b not found: ID does not exist"
Sep 11 04:18:43.238806 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.238742    2582 scope.go:117] "RemoveContainer" containerID="35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd"
Sep 11 04:18:43.247861 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.247806    2582 scope.go:117] "RemoveContainer" containerID="35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd"
Sep 11 04:18:43.248173 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:18:43.248143    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd\": container with ID starting with 35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd not found: ID does not exist" containerID="35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd"
Sep 11 04:18:43.248258 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.248184    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd"} err="failed to get container status \"35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd\": rpc error: code = NotFound desc = could not find container \"35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd\": container with ID starting with 35e76f9eca24fe445a24c4a0975813a2c97f59a2614672bbc18b104cffe4f3cd not found: ID does not exist"
Sep 11 04:18:43.250466 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.250444    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"]
Sep 11 04:18:43.254319 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.254294    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bf30f8-79c46ddc6b-kh2fl"]
Sep 11 04:18:43.265234 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.265205    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"]
Sep 11 04:18:43.271706 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:43.271679    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bf30f8-54b84448fc-7m4pg"]
Sep 11 04:18:44.275617 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:44.275583    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbdf18b-7100-43f5-abf7-3eedf013b05c" path="/var/lib/kubelet/pods/4fbdf18b-7100-43f5-abf7-3eedf013b05c/volumes"
Sep 11 04:18:44.275994 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:44.275987    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e8a1711-e42e-4d48-90d8-b96d9ff50160" path="/var/lib/kubelet/pods/5e8a1711-e42e-4d48-90d8-b96d9ff50160/volumes"
Sep 11 04:18:44.276283 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:44.276271    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c" path="/var/lib/kubelet/pods/d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c/volumes"
Sep 11 04:18:46.326600 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:46.326483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:18:46.346031 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:46.329333    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:18:46.508917 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:46.508879    2582 scope.go:117] "RemoveContainer" containerID="5e7dd5be87b69be0b0625b2af5207daee577d687fa579d5a852a8ba760805bea"
Sep 11 04:18:49.572326 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.572286    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"]
Sep 11 04:18:49.573903 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.573874    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" containerName="main"
Sep 11 04:18:49.574060 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574049    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" containerName="main"
Sep 11 04:18:49.574140 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574131    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c" containerName="revoke-keys"
Sep 11 04:18:49.574210 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574202    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c" containerName="revoke-keys"
Sep 11 04:18:49.574305 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574296    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8a1711-e42e-4d48-90d8-b96d9ff50160" containerName="payload-processing"
Sep 11 04:18:49.574383 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574366    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8a1711-e42e-4d48-90d8-b96d9ff50160" containerName="payload-processing"
Sep 11 04:18:49.574470 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574459    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbdf18b-7100-43f5-abf7-3eedf013b05c" containerName="payload-pre-processing"
Sep 11 04:18:49.574545 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574536    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbdf18b-7100-43f5-abf7-3eedf013b05c" containerName="payload-pre-processing"
Sep 11 04:18:49.574872 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574856    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbdf18b-7100-43f5-abf7-3eedf013b05c" containerName="payload-pre-processing"
Sep 11 04:18:49.574979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.574969    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6145a7ac-12f2-4bc6-abad-cfe5143f1d9e" containerName="main"
Sep 11 04:18:49.575066 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.575056    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8a1711-e42e-4d48-90d8-b96d9ff50160" containerName="payload-processing"
Sep 11 04:18:49.575146 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.575138    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66b1c8e-3ba4-4d62-94a2-1dbb61b6547c" containerName="revoke-keys"
Sep 11 04:18:49.581027 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.580999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"]
Sep 11 04:18:49.581159 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.581111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"
Sep 11 04:18:49.583960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.583925    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:18:49.584086 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.584028    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:18:49.665263 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.665220    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3938d93-e344-4e94-9eb7-82b42a8f4ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-599baf-v46xd\" (UID: \"b3938d93-e344-4e94-9eb7-82b42a8f4ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"
Sep 11 04:18:49.766145 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.766108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3938d93-e344-4e94-9eb7-82b42a8f4ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-599baf-v46xd\" (UID: \"b3938d93-e344-4e94-9eb7-82b42a8f4ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"
Sep 11 04:18:49.766789 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.766770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3938d93-e344-4e94-9eb7-82b42a8f4ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-599baf-v46xd\" (UID: \"b3938d93-e344-4e94-9eb7-82b42a8f4ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"
Sep 11 04:18:49.892744 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:49.892657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"
Sep 11 04:18:50.166342 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:50.166317    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"]
Sep 11 04:18:50.168396 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:18:50.168363    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3938d93_e344_4e94_9eb7_82b42a8f4ea5.slice/crio-90a236dac97601814edb05438720224fa5d911700cb69c61102a83111121b009 WatchSource:0}: Error finding container 90a236dac97601814edb05438720224fa5d911700cb69c61102a83111121b009: Status 404 returned error can't find the container with id 90a236dac97601814edb05438720224fa5d911700cb69c61102a83111121b009
Sep 11 04:18:50.275953 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:50.275918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd" event={"ID":"b3938d93-e344-4e94-9eb7-82b42a8f4ea5","Type":"ContainerStarted","Data":"90a236dac97601814edb05438720224fa5d911700cb69c61102a83111121b009"}
Sep 11 04:18:51.283189 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:51.283152    2582 generic.go:356] "Generic (PLEG): container finished" podID="b3938d93-e344-4e94-9eb7-82b42a8f4ea5" containerID="43cc5c5a364fdf439b3e19706d019fb4e87640c2b7d61cbaef438a7f7ffdc24e" exitCode=0
Sep 11 04:18:51.283643 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:51.283242    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd" event={"ID":"b3938d93-e344-4e94-9eb7-82b42a8f4ea5","Type":"ContainerDied","Data":"43cc5c5a364fdf439b3e19706d019fb4e87640c2b7d61cbaef438a7f7ffdc24e"}
Sep 11 04:18:52.425170 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:52.425143    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"
Sep 11 04:18:52.594692 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:52.594595    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3938d93-e344-4e94-9eb7-82b42a8f4ea5-maas-api-service-ca\") pod \"b3938d93-e344-4e94-9eb7-82b42a8f4ea5\" (UID: \"b3938d93-e344-4e94-9eb7-82b42a8f4ea5\") "
Sep 11 04:18:52.594958 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:52.594931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3938d93-e344-4e94-9eb7-82b42a8f4ea5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3938d93-e344-4e94-9eb7-82b42a8f4ea5" (UID: "b3938d93-e344-4e94-9eb7-82b42a8f4ea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:18:52.696141 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:52.696103    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3938d93-e344-4e94-9eb7-82b42a8f4ea5-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:18:53.297200 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:53.297160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd" event={"ID":"b3938d93-e344-4e94-9eb7-82b42a8f4ea5","Type":"ContainerDied","Data":"90a236dac97601814edb05438720224fa5d911700cb69c61102a83111121b009"}
Sep 11 04:18:53.297200 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:53.297203    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"
Sep 11 04:18:53.297429 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:53.297228    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a236dac97601814edb05438720224fa5d911700cb69c61102a83111121b009"
Sep 11 04:18:56.696571 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.696507    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"]
Sep 11 04:18:56.697068 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.697022    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3938d93-e344-4e94-9eb7-82b42a8f4ea5" containerName="revoke-keys"
Sep 11 04:18:56.697068 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.697037    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3938d93-e344-4e94-9eb7-82b42a8f4ea5" containerName="revoke-keys"
Sep 11 04:18:56.697161 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.697114    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3938d93-e344-4e94-9eb7-82b42a8f4ea5" containerName="revoke-keys"
Sep 11 04:18:56.700445 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.700426    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.703145 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.703116    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"openshift-service-ca.crt\""
Sep 11 04:18:56.703265 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.703118    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"kube-root-ca.crt\""
Sep 11 04:18:56.704264 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.704240    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"rate-test-model-9342ed-kserve-self-signed-certs\""
Sep 11 04:18:56.704264 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.704254    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"default-dockercfg-552lm\""
Sep 11 04:18:56.710413 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.710392    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"]
Sep 11 04:18:56.838674 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.838634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-dshm\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.838861 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.838687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-tmp-dir\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.838861 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.838778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdzq\" (UniqueName: \"kubernetes.io/projected/9408dff7-9e24-4685-b81c-204cc54b2b53-kube-api-access-nrdzq\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.838861 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.838819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-model-cache\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.839038 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.838947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9408dff7-9e24-4685-b81c-204cc54b2b53-tls-certs\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.839096 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.839036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-home\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdzq\" (UniqueName: \"kubernetes.io/projected/9408dff7-9e24-4685-b81c-204cc54b2b53-kube-api-access-nrdzq\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-model-cache\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940679 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9408dff7-9e24-4685-b81c-204cc54b2b53-tls-certs\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940679 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-home\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940679 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-dshm\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940679 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-tmp-dir\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940931 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-home\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.940931 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940923    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-tmp-dir\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.941095 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.940954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-model-cache\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.942860 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.942840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-dshm\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.943044 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.943025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9408dff7-9e24-4685-b81c-204cc54b2b53-tls-certs\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:56.948499 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:56.948439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdzq\" (UniqueName: \"kubernetes.io/projected/9408dff7-9e24-4685-b81c-204cc54b2b53-kube-api-access-nrdzq\") pod \"rate-test-model-9342ed-kserve-6fcc44646c-bn5hk\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:57.011151 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:57.011110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:18:57.286209 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:57.286182    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"]
Sep 11 04:18:57.287951 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:18:57.287924    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9408dff7_9e24_4685_b81c_204cc54b2b53.slice/crio-dba2042fecffdb09e6a5bcb1b820729e8ad91face807216a50b666f7ca9547cf WatchSource:0}: Error finding container dba2042fecffdb09e6a5bcb1b820729e8ad91face807216a50b666f7ca9547cf: Status 404 returned error can't find the container with id dba2042fecffdb09e6a5bcb1b820729e8ad91face807216a50b666f7ca9547cf
Sep 11 04:18:57.320232 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:57.320195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk" event={"ID":"9408dff7-9e24-4685-b81c-204cc54b2b53","Type":"ContainerStarted","Data":"dba2042fecffdb09e6a5bcb1b820729e8ad91face807216a50b666f7ca9547cf"}
Sep 11 04:18:58.327648 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:58.327598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk" event={"ID":"9408dff7-9e24-4685-b81c-204cc54b2b53","Type":"ContainerStarted","Data":"8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da"}
Sep 11 04:18:58.346889 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:18:58.346833    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk" podStartSLOduration=2.346816867 podStartE2EDuration="2.346816867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:18:58.344498139 +0000 UTC m=+1512.608976510" watchObservedRunningTime="2026-09-11 04:18:58.346816867 +0000 UTC m=+1512.611295265"
Sep 11 04:19:05.449762 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.449725    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"]
Sep 11 04:19:05.450285 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.450007    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4" podUID="3e637b70-180e-4ee4-b942-20a26cffb917" containerName="payload-processing" containerID="cri-o://759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4" gracePeriod=30
Sep 11 04:19:05.465684 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.465653    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"]
Sep 11 04:19:05.466088 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.466057    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8" podUID="b513424e-0955-4d55-9331-6e7ef6d3a8ea" containerName="payload-pre-processing" containerID="cri-o://ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b" gracePeriod=30
Sep 11 04:19:05.622761 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.622674    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"]
Sep 11 04:19:05.625509 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.625483    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-599baf-v46xd"]
Sep 11 04:19:05.727228 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.727203    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:19:05.753110 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.753082    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:19:05.819816 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.819780    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b513424e-0955-4d55-9331-6e7ef6d3a8ea-plugins-config-volume\") pod \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") "
Sep 11 04:19:05.819991 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.819854    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htxdl\" (UniqueName: \"kubernetes.io/projected/3e637b70-180e-4ee4-b942-20a26cffb917-kube-api-access-htxdl\") pod \"3e637b70-180e-4ee4-b942-20a26cffb917\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") "
Sep 11 04:19:05.819991 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.819929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e637b70-180e-4ee4-b942-20a26cffb917-plugins-config-volume\") pod \"3e637b70-180e-4ee4-b942-20a26cffb917\" (UID: \"3e637b70-180e-4ee4-b942-20a26cffb917\") "
Sep 11 04:19:05.819991 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.819977    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm56w\" (UniqueName: \"kubernetes.io/projected/b513424e-0955-4d55-9331-6e7ef6d3a8ea-kube-api-access-lm56w\") pod \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\" (UID: \"b513424e-0955-4d55-9331-6e7ef6d3a8ea\") "
Sep 11 04:19:05.820253 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.820223    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b513424e-0955-4d55-9331-6e7ef6d3a8ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b513424e-0955-4d55-9331-6e7ef6d3a8ea" (UID: "b513424e-0955-4d55-9331-6e7ef6d3a8ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:19:05.820311 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.820248    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e637b70-180e-4ee4-b942-20a26cffb917-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e637b70-180e-4ee4-b942-20a26cffb917" (UID: "3e637b70-180e-4ee4-b942-20a26cffb917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:19:05.822002 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.821977    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e637b70-180e-4ee4-b942-20a26cffb917-kube-api-access-htxdl" (OuterVolumeSpecName: "kube-api-access-htxdl") pod "3e637b70-180e-4ee4-b942-20a26cffb917" (UID: "3e637b70-180e-4ee4-b942-20a26cffb917"). InnerVolumeSpecName "kube-api-access-htxdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:19:05.822087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.822034    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b513424e-0955-4d55-9331-6e7ef6d3a8ea-kube-api-access-lm56w" (OuterVolumeSpecName: "kube-api-access-lm56w") pod "b513424e-0955-4d55-9331-6e7ef6d3a8ea" (UID: "b513424e-0955-4d55-9331-6e7ef6d3a8ea"). InnerVolumeSpecName "kube-api-access-lm56w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:19:05.921448 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.921393    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e637b70-180e-4ee4-b942-20a26cffb917-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:05.921448 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.921448    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm56w\" (UniqueName: \"kubernetes.io/projected/b513424e-0955-4d55-9331-6e7ef6d3a8ea-kube-api-access-lm56w\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:05.921746 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.921466    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b513424e-0955-4d55-9331-6e7ef6d3a8ea-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:05.921746 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:05.921485    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxdl\" (UniqueName: \"kubernetes.io/projected/3e637b70-180e-4ee4-b942-20a26cffb917-kube-api-access-htxdl\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:06.275988 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.275953    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3938d93-e344-4e94-9eb7-82b42a8f4ea5" path="/var/lib/kubelet/pods/b3938d93-e344-4e94-9eb7-82b42a8f4ea5/volumes"
Sep 11 04:19:06.377909 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.377869    2582 generic.go:356] "Generic (PLEG): container finished" podID="3e637b70-180e-4ee4-b942-20a26cffb917" containerID="759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4" exitCode=0
Sep 11 04:19:06.378087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.377925    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"
Sep 11 04:19:06.378087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.377967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4" event={"ID":"3e637b70-180e-4ee4-b942-20a26cffb917","Type":"ContainerDied","Data":"759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4"}
Sep 11 04:19:06.378087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.378033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4" event={"ID":"3e637b70-180e-4ee4-b942-20a26cffb917","Type":"ContainerDied","Data":"b213f0b2fddc5cc6593518d8e6485b5e8adf80bec9bd1429955739da869987c1"}
Sep 11 04:19:06.378087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.378062    2582 scope.go:117] "RemoveContainer" containerID="759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4"
Sep 11 04:19:06.379581 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.379533    2582 generic.go:356] "Generic (PLEG): container finished" podID="b513424e-0955-4d55-9331-6e7ef6d3a8ea" containerID="ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b" exitCode=0
Sep 11 04:19:06.379692 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.379594    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"
Sep 11 04:19:06.379692 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.379591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8" event={"ID":"b513424e-0955-4d55-9331-6e7ef6d3a8ea","Type":"ContainerDied","Data":"ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b"}
Sep 11 04:19:06.379783 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.379734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8" event={"ID":"b513424e-0955-4d55-9331-6e7ef6d3a8ea","Type":"ContainerDied","Data":"c9af06c941ee88da82813f32747e04fbcbc49e434730fc474118f6074c5c9cf3"}
Sep 11 04:19:06.387643 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.387627    2582 scope.go:117] "RemoveContainer" containerID="759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4"
Sep 11 04:19:06.387971 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:19:06.387946    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4\": container with ID starting with 759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4 not found: ID does not exist" containerID="759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4"
Sep 11 04:19:06.388022 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.387981    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4"} err="failed to get container status \"759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4\": rpc error: code = NotFound desc = could not find container \"759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4\": container with ID starting with 759dd8024bf52a5d2a8171c7419ae8da8f3dfa068dd89795da34dd0ee4d024e4 not found: ID does not exist"
Sep 11 04:19:06.388022 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.387998    2582 scope.go:117] "RemoveContainer" containerID="ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b"
Sep 11 04:19:06.397739 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.397708    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"]
Sep 11 04:19:06.397974 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.397954    2582 scope.go:117] "RemoveContainer" containerID="ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b"
Sep 11 04:19:06.398305 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:19:06.398287    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b\": container with ID starting with ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b not found: ID does not exist" containerID="ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b"
Sep 11 04:19:06.398350 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.398313    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b"} err="failed to get container status \"ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b\": rpc error: code = NotFound desc = could not find container \"ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b\": container with ID starting with ecc5796295894d6996811e8eb387a1a6109436affeaf774aeb894f33297ba92b not found: ID does not exist"
Sep 11 04:19:06.404391 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.404367    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-599baf-77b55f68f-qpnf4"]
Sep 11 04:19:06.415144 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.415113    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"]
Sep 11 04:19:06.421885 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:06.421858    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-599baf-6db59cc666-vcts8"]
Sep 11 04:19:07.012068 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:07.012032    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:19:07.012544 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:07.012089    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:19:07.025260 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:07.025234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:19:07.398981 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:07.398893    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:19:08.274718 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:08.274644    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e637b70-180e-4ee4-b942-20a26cffb917" path="/var/lib/kubelet/pods/3e637b70-180e-4ee4-b942-20a26cffb917/volumes"
Sep 11 04:19:08.275094 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:08.275079    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b513424e-0955-4d55-9331-6e7ef6d3a8ea" path="/var/lib/kubelet/pods/b513424e-0955-4d55-9331-6e7ef6d3a8ea/volumes"
Sep 11 04:19:12.718852 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.718808    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"]
Sep 11 04:19:12.719315 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.719301    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e637b70-180e-4ee4-b942-20a26cffb917" containerName="payload-processing"
Sep 11 04:19:12.719361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.719317    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e637b70-180e-4ee4-b942-20a26cffb917" containerName="payload-processing"
Sep 11 04:19:12.719361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.719329    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b513424e-0955-4d55-9331-6e7ef6d3a8ea" containerName="payload-pre-processing"
Sep 11 04:19:12.719361 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.719334    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b513424e-0955-4d55-9331-6e7ef6d3a8ea" containerName="payload-pre-processing"
Sep 11 04:19:12.719459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.719407    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b513424e-0955-4d55-9331-6e7ef6d3a8ea" containerName="payload-pre-processing"
Sep 11 04:19:12.719459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.719418    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e637b70-180e-4ee4-b942-20a26cffb917" containerName="payload-processing"
Sep 11 04:19:12.724001 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.723982    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"
Sep 11 04:19:12.727267 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.727236    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:19:12.727411 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.727346    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:19:12.729450 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.729423    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"]
Sep 11 04:19:12.789584 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.789532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99286523-43ba-4196-ad57-7c5216442410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm\" (UID: \"99286523-43ba-4196-ad57-7c5216442410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"
Sep 11 04:19:12.891180 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.891137    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99286523-43ba-4196-ad57-7c5216442410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm\" (UID: \"99286523-43ba-4196-ad57-7c5216442410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"
Sep 11 04:19:12.891798 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:12.891776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99286523-43ba-4196-ad57-7c5216442410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm\" (UID: \"99286523-43ba-4196-ad57-7c5216442410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"
Sep 11 04:19:13.036234 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:13.036125    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"
Sep 11 04:19:13.312682 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:13.312652    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"]
Sep 11 04:19:13.313881 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:19:13.313853    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99286523_43ba_4196_ad57_7c5216442410.slice/crio-e88c82c2290f4981950175b86ca6019a2af5cc3f16637b1a502c52f539727cd1 WatchSource:0}: Error finding container e88c82c2290f4981950175b86ca6019a2af5cc3f16637b1a502c52f539727cd1: Status 404 returned error can't find the container with id e88c82c2290f4981950175b86ca6019a2af5cc3f16637b1a502c52f539727cd1
Sep 11 04:19:13.422024 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:13.421983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm" event={"ID":"99286523-43ba-4196-ad57-7c5216442410","Type":"ContainerStarted","Data":"e88c82c2290f4981950175b86ca6019a2af5cc3f16637b1a502c52f539727cd1"}
Sep 11 04:19:14.429065 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:14.429032    2582 generic.go:356] "Generic (PLEG): container finished" podID="99286523-43ba-4196-ad57-7c5216442410" containerID="3568234edaa3125288201b285edf1ed43b1f2d6c0b0be1b2ae3fa51b355f6e75" exitCode=0
Sep 11 04:19:14.429466 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:14.429072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm" event={"ID":"99286523-43ba-4196-ad57-7c5216442410","Type":"ContainerDied","Data":"3568234edaa3125288201b285edf1ed43b1f2d6c0b0be1b2ae3fa51b355f6e75"}
Sep 11 04:19:15.572837 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:15.572808    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"
Sep 11 04:19:15.624651 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:15.624616    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99286523-43ba-4196-ad57-7c5216442410-maas-api-service-ca\") pod \"99286523-43ba-4196-ad57-7c5216442410\" (UID: \"99286523-43ba-4196-ad57-7c5216442410\") "
Sep 11 04:19:15.625036 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:15.625014    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99286523-43ba-4196-ad57-7c5216442410-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99286523-43ba-4196-ad57-7c5216442410" (UID: "99286523-43ba-4196-ad57-7c5216442410"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:19:15.725660 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:15.725537    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99286523-43ba-4196-ad57-7c5216442410-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:16.441873 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:16.441845    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"
Sep 11 04:19:16.442049 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:16.441887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm" event={"ID":"99286523-43ba-4196-ad57-7c5216442410","Type":"ContainerDied","Data":"e88c82c2290f4981950175b86ca6019a2af5cc3f16637b1a502c52f539727cd1"}
Sep 11 04:19:16.442049 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:16.441941    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88c82c2290f4981950175b86ca6019a2af5cc3f16637b1a502c52f539727cd1"
Sep 11 04:19:23.785082 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:23.785044    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"]
Sep 11 04:19:23.787883 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:23.787858    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-772e41-29818335-dldlq"]
Sep 11 04:19:23.824181 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:23.824139    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"]
Sep 11 04:19:23.824479 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:23.824450    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4" podUID="daee6857-3463-47b3-a162-ab7ff586ef74" containerName="payload-pre-processing" containerID="cri-o://34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8" gracePeriod=30
Sep 11 04:19:23.991018 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:23.990983    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"]
Sep 11 04:19:23.998410 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:23.998382    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-772e41-xhkjm"]
Sep 11 04:19:24.085677 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.085650    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:19:24.212851 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.212819    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/daee6857-3463-47b3-a162-ab7ff586ef74-kube-api-access-vdsdb\") pod \"daee6857-3463-47b3-a162-ab7ff586ef74\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") "
Sep 11 04:19:24.213040 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.213020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee6857-3463-47b3-a162-ab7ff586ef74-plugins-config-volume\") pod \"daee6857-3463-47b3-a162-ab7ff586ef74\" (UID: \"daee6857-3463-47b3-a162-ab7ff586ef74\") "
Sep 11 04:19:24.213374 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.213344    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daee6857-3463-47b3-a162-ab7ff586ef74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "daee6857-3463-47b3-a162-ab7ff586ef74" (UID: "daee6857-3463-47b3-a162-ab7ff586ef74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:19:24.215017 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.214996    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daee6857-3463-47b3-a162-ab7ff586ef74-kube-api-access-vdsdb" (OuterVolumeSpecName: "kube-api-access-vdsdb") pod "daee6857-3463-47b3-a162-ab7ff586ef74" (UID: "daee6857-3463-47b3-a162-ab7ff586ef74"). InnerVolumeSpecName "kube-api-access-vdsdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:19:24.275967 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.275932    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b744888-36e6-4b6c-823a-808edf199c49" path="/var/lib/kubelet/pods/5b744888-36e6-4b6c-823a-808edf199c49/volumes"
Sep 11 04:19:24.276283 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.276270    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99286523-43ba-4196-ad57-7c5216442410" path="/var/lib/kubelet/pods/99286523-43ba-4196-ad57-7c5216442410/volumes"
Sep 11 04:19:24.315000 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.314906    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daee6857-3463-47b3-a162-ab7ff586ef74-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:24.315000 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.314943    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/daee6857-3463-47b3-a162-ab7ff586ef74-kube-api-access-vdsdb\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:24.489032 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.488995    2582 generic.go:356] "Generic (PLEG): container finished" podID="daee6857-3463-47b3-a162-ab7ff586ef74" containerID="34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8" exitCode=0
Sep 11 04:19:24.489220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.489037    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"
Sep 11 04:19:24.489220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.489047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4" event={"ID":"daee6857-3463-47b3-a162-ab7ff586ef74","Type":"ContainerDied","Data":"34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8"}
Sep 11 04:19:24.489220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.489106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4" event={"ID":"daee6857-3463-47b3-a162-ab7ff586ef74","Type":"ContainerDied","Data":"979fb6c62bfd3448b335cf02d07fb8efafc598f3e5284304f8e18b0519be6cdc"}
Sep 11 04:19:24.489220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.489135    2582 scope.go:117] "RemoveContainer" containerID="34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8"
Sep 11 04:19:24.498462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.498443    2582 scope.go:117] "RemoveContainer" containerID="34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8"
Sep 11 04:19:24.498780 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:19:24.498757    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8\": container with ID starting with 34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8 not found: ID does not exist" containerID="34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8"
Sep 11 04:19:24.498852 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.498789    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8"} err="failed to get container status \"34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8\": rpc error: code = NotFound desc = could not find container \"34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8\": container with ID starting with 34f257cec475f1aa6162b3fb25ef7cabc0a4adaf6e63ab630bc68e0c5ed980a8 not found: ID does not exist"
Sep 11 04:19:24.510674 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.510645    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"]
Sep 11 04:19:24.515025 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:24.515000    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-772e41-665579d78-8zst4"]
Sep 11 04:19:26.276439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:26.276406    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daee6857-3463-47b3-a162-ab7ff586ef74" path="/var/lib/kubelet/pods/daee6857-3463-47b3-a162-ab7ff586ef74/volumes"
Sep 11 04:19:30.683995 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.683956    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"]
Sep 11 04:19:30.684411 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.684398    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99286523-43ba-4196-ad57-7c5216442410" containerName="revoke-keys"
Sep 11 04:19:30.684411 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.684412    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99286523-43ba-4196-ad57-7c5216442410" containerName="revoke-keys"
Sep 11 04:19:30.684503 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.684427    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daee6857-3463-47b3-a162-ab7ff586ef74" containerName="payload-pre-processing"
Sep 11 04:19:30.684503 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.684433    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="daee6857-3463-47b3-a162-ab7ff586ef74" containerName="payload-pre-processing"
Sep 11 04:19:30.684503 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.684498    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="99286523-43ba-4196-ad57-7c5216442410" containerName="revoke-keys"
Sep 11 04:19:30.684628 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.684506    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="daee6857-3463-47b3-a162-ab7ff586ef74" containerName="payload-pre-processing"
Sep 11 04:19:30.688972 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.688947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"
Sep 11 04:19:30.692318 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.692272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:19:30.692473 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.692313    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:19:30.695290 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.695260    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"]
Sep 11 04:19:30.786897 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.786864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88bf87fa-8efe-436d-b5eb-fb610f1448f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9\" (UID: \"88bf87fa-8efe-436d-b5eb-fb610f1448f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"
Sep 11 04:19:30.888440 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.888398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88bf87fa-8efe-436d-b5eb-fb610f1448f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9\" (UID: \"88bf87fa-8efe-436d-b5eb-fb610f1448f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"
Sep 11 04:19:30.889087 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:30.889063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88bf87fa-8efe-436d-b5eb-fb610f1448f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9\" (UID: \"88bf87fa-8efe-436d-b5eb-fb610f1448f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"
Sep 11 04:19:31.002518 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:31.002428    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"
Sep 11 04:19:31.276736 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:31.276707    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"]
Sep 11 04:19:31.278866 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:19:31.278830    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bf87fa_8efe_436d_b5eb_fb610f1448f5.slice/crio-2a546802c262fbbe640e7614238c0f5d81c2caf5d4788febd902a0556cdd7010 WatchSource:0}: Error finding container 2a546802c262fbbe640e7614238c0f5d81c2caf5d4788febd902a0556cdd7010: Status 404 returned error can't find the container with id 2a546802c262fbbe640e7614238c0f5d81c2caf5d4788febd902a0556cdd7010
Sep 11 04:19:31.529898 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:31.529808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9" event={"ID":"88bf87fa-8efe-436d-b5eb-fb610f1448f5","Type":"ContainerStarted","Data":"2a546802c262fbbe640e7614238c0f5d81c2caf5d4788febd902a0556cdd7010"}
Sep 11 04:19:32.537002 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:32.536965    2582 generic.go:356] "Generic (PLEG): container finished" podID="88bf87fa-8efe-436d-b5eb-fb610f1448f5" containerID="303dbbb3d619c7c5f1d195d714055625c592a20863366305056f5f9569dc4dc7" exitCode=0
Sep 11 04:19:32.537452 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:32.537051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9" event={"ID":"88bf87fa-8efe-436d-b5eb-fb610f1448f5","Type":"ContainerDied","Data":"303dbbb3d619c7c5f1d195d714055625c592a20863366305056f5f9569dc4dc7"}
Sep 11 04:19:33.686452 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:33.686427    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"
Sep 11 04:19:33.717671 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:33.717637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88bf87fa-8efe-436d-b5eb-fb610f1448f5-maas-api-service-ca\") pod \"88bf87fa-8efe-436d-b5eb-fb610f1448f5\" (UID: \"88bf87fa-8efe-436d-b5eb-fb610f1448f5\") "
Sep 11 04:19:33.717973 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:33.717947    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bf87fa-8efe-436d-b5eb-fb610f1448f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88bf87fa-8efe-436d-b5eb-fb610f1448f5" (UID: "88bf87fa-8efe-436d-b5eb-fb610f1448f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:19:33.818988 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:33.818899    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88bf87fa-8efe-436d-b5eb-fb610f1448f5-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:34.549136 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:34.549040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9" event={"ID":"88bf87fa-8efe-436d-b5eb-fb610f1448f5","Type":"ContainerDied","Data":"2a546802c262fbbe640e7614238c0f5d81c2caf5d4788febd902a0556cdd7010"}
Sep 11 04:19:34.549136 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:34.549069    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"
Sep 11 04:19:34.549136 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:34.549092    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a546802c262fbbe640e7614238c0f5d81c2caf5d4788febd902a0556cdd7010"
Sep 11 04:19:38.081182 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:38.081145    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"]
Sep 11 04:19:41.184719 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.184684    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"]
Sep 11 04:19:41.185275 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.185141    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp" podUID="68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" containerName="payload-processing" containerID="cri-o://1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d" gracePeriod=30
Sep 11 04:19:41.412506 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.412464    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"]
Sep 11 04:19:41.416051 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.416026    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f8229c-zhml9"]
Sep 11 04:19:41.448737 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.448670    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:19:41.488982 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.488949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dtvn\" (UniqueName: \"kubernetes.io/projected/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-kube-api-access-8dtvn\") pod \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") "
Sep 11 04:19:41.489167 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.489005    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-plugins-config-volume\") pod \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\" (UID: \"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5\") "
Sep 11 04:19:41.489449 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.489426    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" (UID: "68fbfcc0-fa18-41be-a26d-85dbfe34e0a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:19:41.491171 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.491147    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-kube-api-access-8dtvn" (OuterVolumeSpecName: "kube-api-access-8dtvn") pod "68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" (UID: "68fbfcc0-fa18-41be-a26d-85dbfe34e0a5"). InnerVolumeSpecName "kube-api-access-8dtvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:19:41.590913 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.590876    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dtvn\" (UniqueName: \"kubernetes.io/projected/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-kube-api-access-8dtvn\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:41.590913 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.590911    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:41.593161 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.593130    2582 generic.go:356] "Generic (PLEG): container finished" podID="68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" containerID="1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d" exitCode=0
Sep 11 04:19:41.593324 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.593183    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"
Sep 11 04:19:41.593324 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.593215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp" event={"ID":"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5","Type":"ContainerDied","Data":"1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d"}
Sep 11 04:19:41.593324 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.593271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp" event={"ID":"68fbfcc0-fa18-41be-a26d-85dbfe34e0a5","Type":"ContainerDied","Data":"60bf3750a0ed7306e73fa460f38b91028d6f039b9ed3fb93d1d0166a6312db72"}
Sep 11 04:19:41.593324 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.593293    2582 scope.go:117] "RemoveContainer" containerID="1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d"
Sep 11 04:19:41.602809 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.602787    2582 scope.go:117] "RemoveContainer" containerID="1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d"
Sep 11 04:19:41.603121 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:19:41.603101    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d\": container with ID starting with 1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d not found: ID does not exist" containerID="1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d"
Sep 11 04:19:41.603177 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.603131    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d"} err="failed to get container status \"1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d\": rpc error: code = NotFound desc = could not find container \"1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d\": container with ID starting with 1b003b7f6f105ac284170daa48beb1ee86afeaec7e3b404b2be376eb00de0e0d not found: ID does not exist"
Sep 11 04:19:41.614863 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.614836    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"]
Sep 11 04:19:41.618628 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:41.618603    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8229c-8b74878d7-z82qp"]
Sep 11 04:19:42.275734 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:42.275702    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" path="/var/lib/kubelet/pods/68fbfcc0-fa18-41be-a26d-85dbfe34e0a5/volumes"
Sep 11 04:19:42.276121 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:42.276056    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88bf87fa-8efe-436d-b5eb-fb610f1448f5" path="/var/lib/kubelet/pods/88bf87fa-8efe-436d-b5eb-fb610f1448f5/volumes"
Sep 11 04:19:44.206365 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.206329    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"]
Sep 11 04:19:44.206843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.206825    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" containerName="payload-processing"
Sep 11 04:19:44.206889 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.206844    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" containerName="payload-processing"
Sep 11 04:19:44.206889 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.206865    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bf87fa-8efe-436d-b5eb-fb610f1448f5" containerName="revoke-keys"
Sep 11 04:19:44.206889 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.206875    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bf87fa-8efe-436d-b5eb-fb610f1448f5" containerName="revoke-keys"
Sep 11 04:19:44.206986 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.206964    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fbfcc0-fa18-41be-a26d-85dbfe34e0a5" containerName="payload-processing"
Sep 11 04:19:44.206986 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.206973    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bf87fa-8efe-436d-b5eb-fb610f1448f5" containerName="revoke-keys"
Sep 11 04:19:44.211524 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.211505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.214096 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.214070    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fdc690\""
Sep 11 04:19:44.214229 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.214074    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fdc690-dockercfg-h5xsr\""
Sep 11 04:19:44.220477 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.220453    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"]
Sep 11 04:19:44.239921 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.239894    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"]
Sep 11 04:19:44.243478 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.243460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.254247 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.254218    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"]
Sep 11 04:19:44.319309 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.319258    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9qn\" (UniqueName: \"kubernetes.io/projected/fb936454-d315-4b34-a7a2-857ab57efb75-kube-api-access-5s9qn\") pod \"payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.319499 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.319340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb936454-d315-4b34-a7a2-857ab57efb75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.319499 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.319389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzgj\" (UniqueName: \"kubernetes.io/projected/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-kube-api-access-jqzgj\") pod \"payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.319632 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.319497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.420526 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.420475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.420733 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.420580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9qn\" (UniqueName: \"kubernetes.io/projected/fb936454-d315-4b34-a7a2-857ab57efb75-kube-api-access-5s9qn\") pod \"payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.420733 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.420613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb936454-d315-4b34-a7a2-857ab57efb75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.420733 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.420635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzgj\" (UniqueName: \"kubernetes.io/projected/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-kube-api-access-jqzgj\") pod \"payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.421231 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.421211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.421366 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.421349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb936454-d315-4b34-a7a2-857ab57efb75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.428799 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.428772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9qn\" (UniqueName: \"kubernetes.io/projected/fb936454-d315-4b34-a7a2-857ab57efb75-kube-api-access-5s9qn\") pod \"payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.428926 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.428812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzgj\" (UniqueName: \"kubernetes.io/projected/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-kube-api-access-jqzgj\") pod \"payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.523196 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.523098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:44.556821 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.556788    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:44.799479 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.799449    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"]
Sep 11 04:19:44.801530 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:19:44.801502    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb936454_d315_4b34_a7a2_857ab57efb75.slice/crio-4b64f44e33f680f91ee2e420c937ae7d4ff41a4fe698764d7931088c9478dfed WatchSource:0}: Error finding container 4b64f44e33f680f91ee2e420c937ae7d4ff41a4fe698764d7931088c9478dfed: Status 404 returned error can't find the container with id 4b64f44e33f680f91ee2e420c937ae7d4ff41a4fe698764d7931088c9478dfed
Sep 11 04:19:44.843333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:44.843311    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"]
Sep 11 04:19:44.844127 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:19:44.844102    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cb6617_fe7e_4297_98c0_97c7668cd1fc.slice/crio-ee9e38755d4f8d1aaca8ac6dbfcd771544ebb6a42bd4bbbff6b59070d6e89da3 WatchSource:0}: Error finding container ee9e38755d4f8d1aaca8ac6dbfcd771544ebb6a42bd4bbbff6b59070d6e89da3: Status 404 returned error can't find the container with id ee9e38755d4f8d1aaca8ac6dbfcd771544ebb6a42bd4bbbff6b59070d6e89da3
Sep 11 04:19:45.629665 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.629612    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4" event={"ID":"d4cb6617-fe7e-4297-98c0-97c7668cd1fc","Type":"ContainerStarted","Data":"a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef"}
Sep 11 04:19:45.630151 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.629688    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:19:45.630151 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.629714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4" event={"ID":"d4cb6617-fe7e-4297-98c0-97c7668cd1fc","Type":"ContainerStarted","Data":"ee9e38755d4f8d1aaca8ac6dbfcd771544ebb6a42bd4bbbff6b59070d6e89da3"}
Sep 11 04:19:45.631617 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.631587    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs" event={"ID":"fb936454-d315-4b34-a7a2-857ab57efb75","Type":"ContainerStarted","Data":"3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16"}
Sep 11 04:19:45.631757 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.631631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs" event={"ID":"fb936454-d315-4b34-a7a2-857ab57efb75","Type":"ContainerStarted","Data":"4b64f44e33f680f91ee2e420c937ae7d4ff41a4fe698764d7931088c9478dfed"}
Sep 11 04:19:45.631757 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.631659    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:45.647896 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.647844    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4" podStartSLOduration=1.647828594 podStartE2EDuration="1.647828594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:19:45.64631441 +0000 UTC m=+1559.910792782" watchObservedRunningTime="2026-09-11 04:19:45.647828594 +0000 UTC m=+1559.912306964"
Sep 11 04:19:45.671158 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:45.671102    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs" podStartSLOduration=1.671085127 podStartE2EDuration="1.671085127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:19:45.669732214 +0000 UTC m=+1559.934210588" watchObservedRunningTime="2026-09-11 04:19:45.671085127 +0000 UTC m=+1559.935563497"
Sep 11 04:19:53.100713 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.100663    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk" podUID="9408dff7-9e24-4685-b81c-204cc54b2b53" containerName="main" containerID="cri-o://8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da" gracePeriod=45
Sep 11 04:19:53.364776 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.364715    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:19:53.515877 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.515834    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrdzq\" (UniqueName: \"kubernetes.io/projected/9408dff7-9e24-4685-b81c-204cc54b2b53-kube-api-access-nrdzq\") pod \"9408dff7-9e24-4685-b81c-204cc54b2b53\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") "
Sep 11 04:19:53.515877 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.515880    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-home\") pod \"9408dff7-9e24-4685-b81c-204cc54b2b53\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") "
Sep 11 04:19:53.516149 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.515974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-tmp-dir\") pod \"9408dff7-9e24-4685-b81c-204cc54b2b53\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") "
Sep 11 04:19:53.516149 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516049    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-dshm\") pod \"9408dff7-9e24-4685-b81c-204cc54b2b53\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") "
Sep 11 04:19:53.516149 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516075    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-model-cache\") pod \"9408dff7-9e24-4685-b81c-204cc54b2b53\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") "
Sep 11 04:19:53.516149 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9408dff7-9e24-4685-b81c-204cc54b2b53-tls-certs\") pod \"9408dff7-9e24-4685-b81c-204cc54b2b53\" (UID: \"9408dff7-9e24-4685-b81c-204cc54b2b53\") "
Sep 11 04:19:53.516358 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516254    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-home" (OuterVolumeSpecName: "home") pod "9408dff7-9e24-4685-b81c-204cc54b2b53" (UID: "9408dff7-9e24-4685-b81c-204cc54b2b53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:19:53.516358 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9408dff7-9e24-4685-b81c-204cc54b2b53" (UID: "9408dff7-9e24-4685-b81c-204cc54b2b53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:19:53.516462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-model-cache" (OuterVolumeSpecName: "model-cache") pod "9408dff7-9e24-4685-b81c-204cc54b2b53" (UID: "9408dff7-9e24-4685-b81c-204cc54b2b53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:19:53.516528 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516510    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-model-cache\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:53.516618 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516530    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-home\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:53.516618 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.516589    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-tmp-dir\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:53.518103 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.518075    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9408dff7-9e24-4685-b81c-204cc54b2b53-kube-api-access-nrdzq" (OuterVolumeSpecName: "kube-api-access-nrdzq") pod "9408dff7-9e24-4685-b81c-204cc54b2b53" (UID: "9408dff7-9e24-4685-b81c-204cc54b2b53"). InnerVolumeSpecName "kube-api-access-nrdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:19:53.518203 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.518162    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-dshm" (OuterVolumeSpecName: "dshm") pod "9408dff7-9e24-4685-b81c-204cc54b2b53" (UID: "9408dff7-9e24-4685-b81c-204cc54b2b53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:19:53.518243 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.518208    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9408dff7-9e24-4685-b81c-204cc54b2b53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9408dff7-9e24-4685-b81c-204cc54b2b53" (UID: "9408dff7-9e24-4685-b81c-204cc54b2b53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:19:53.617981 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.617897    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdzq\" (UniqueName: \"kubernetes.io/projected/9408dff7-9e24-4685-b81c-204cc54b2b53-kube-api-access-nrdzq\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:53.617981 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.617929    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9408dff7-9e24-4685-b81c-204cc54b2b53-dshm\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:53.617981 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.617939    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9408dff7-9e24-4685-b81c-204cc54b2b53-tls-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:19:53.678595 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.678541    2582 generic.go:356] "Generic (PLEG): container finished" podID="9408dff7-9e24-4685-b81c-204cc54b2b53" containerID="8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da" exitCode=0
Sep 11 04:19:53.678595 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.678584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk" event={"ID":"9408dff7-9e24-4685-b81c-204cc54b2b53","Type":"ContainerDied","Data":"8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da"}
Sep 11 04:19:53.678760 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.678602    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"
Sep 11 04:19:53.678760 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.678632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk" event={"ID":"9408dff7-9e24-4685-b81c-204cc54b2b53","Type":"ContainerDied","Data":"dba2042fecffdb09e6a5bcb1b820729e8ad91face807216a50b666f7ca9547cf"}
Sep 11 04:19:53.678760 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.678650    2582 scope.go:117] "RemoveContainer" containerID="8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da"
Sep 11 04:19:53.688308 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.688285    2582 scope.go:117] "RemoveContainer" containerID="8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da"
Sep 11 04:19:53.688612 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:19:53.688591    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da\": container with ID starting with 8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da not found: ID does not exist" containerID="8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da"
Sep 11 04:19:53.688669 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.688624    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da"} err="failed to get container status \"8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da\": rpc error: code = NotFound desc = could not find container \"8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da\": container with ID starting with 8b0ff7c23564cda5dd1eac8dd941b9eda6cb9f9dbc7c32f8e7e9a68348e250da not found: ID does not exist"
Sep 11 04:19:53.700110 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.700085    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"]
Sep 11 04:19:53.705876 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:53.705852    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/rate-test-model-9342ed-kserve-6fcc44646c-bn5hk"]
Sep 11 04:19:54.275123 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:54.275091    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9408dff7-9e24-4685-b81c-204cc54b2b53" path="/var/lib/kubelet/pods/9408dff7-9e24-4685-b81c-204cc54b2b53/volumes"
Sep 11 04:19:56.639748 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:56.639706    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:19:56.640144 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:19:56.639789    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:20:13.230566 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.230513    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"]
Sep 11 04:20:13.231056 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.230976    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9408dff7-9e24-4685-b81c-204cc54b2b53" containerName="main"
Sep 11 04:20:13.231056 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.230988    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9408dff7-9e24-4685-b81c-204cc54b2b53" containerName="main"
Sep 11 04:20:13.231131 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.231069    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9408dff7-9e24-4685-b81c-204cc54b2b53" containerName="main"
Sep 11 04:20:13.235788 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.235761    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.238722 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.238691    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7a5a85-dockercfg-l66p9\""
Sep 11 04:20:13.238827 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.238737    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7a5a85\""
Sep 11 04:20:13.247838 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.247811    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"]
Sep 11 04:20:13.308049 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.307996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvgwz\" (UniqueName: \"kubernetes.io/projected/980aeac0-f71d-4004-be26-9e9125947738-kube-api-access-kvgwz\") pod \"payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.308274 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.308104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980aeac0-f71d-4004-be26-9e9125947738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.409288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.409248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvgwz\" (UniqueName: \"kubernetes.io/projected/980aeac0-f71d-4004-be26-9e9125947738-kube-api-access-kvgwz\") pod \"payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.409288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.409288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980aeac0-f71d-4004-be26-9e9125947738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.409982 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.409960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980aeac0-f71d-4004-be26-9e9125947738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.417372 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.417351    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvgwz\" (UniqueName: \"kubernetes.io/projected/980aeac0-f71d-4004-be26-9e9125947738-kube-api-access-kvgwz\") pod \"payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.548475 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.548389    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:13.822453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:13.822412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"]
Sep 11 04:20:13.823948 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:20:13.823916    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980aeac0_f71d_4004_be26_9e9125947738.slice/crio-0187880b960c9c06fbf291c5967202c0434527bc910f4e53e8820693e6f17729 WatchSource:0}: Error finding container 0187880b960c9c06fbf291c5967202c0434527bc910f4e53e8820693e6f17729: Status 404 returned error can't find the container with id 0187880b960c9c06fbf291c5967202c0434527bc910f4e53e8820693e6f17729
Sep 11 04:20:14.801649 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:14.801613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx" event={"ID":"980aeac0-f71d-4004-be26-9e9125947738","Type":"ContainerStarted","Data":"b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c"}
Sep 11 04:20:14.802097 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:14.801664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx" event={"ID":"980aeac0-f71d-4004-be26-9e9125947738","Type":"ContainerStarted","Data":"0187880b960c9c06fbf291c5967202c0434527bc910f4e53e8820693e6f17729"}
Sep 11 04:20:14.802097 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:14.801727    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:20:14.820934 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:14.820873    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx" podStartSLOduration=1.8208561250000002 podStartE2EDuration="1.820856125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:20:14.818511264 +0000 UTC m=+1589.082989636" watchObservedRunningTime="2026-09-11 04:20:14.820856125 +0000 UTC m=+1589.085334496"
Sep 11 04:20:25.810267 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:20:25.810235    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:21:12.515303 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:12.515266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4_d4cb6617-fe7e-4297-98c0-97c7668cd1fc/payload-processing/0.log"
Sep 11 04:21:18.611851 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.611811    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"]
Sep 11 04:21:18.615568 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.615534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.618178 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.618159    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-84b423\""
Sep 11 04:21:18.618278 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.618216    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-84b423-dockercfg-vx8g6\""
Sep 11 04:21:18.624842 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.624819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"]
Sep 11 04:21:18.668031 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.667992    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"]
Sep 11 04:21:18.671584 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.671542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:18.681971 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.681944    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"]
Sep 11 04:21:18.766169 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.766135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbcd6\" (UniqueName: \"kubernetes.io/projected/b37dbd87-a283-40be-8729-f2ab95c1fa18-kube-api-access-rbcd6\") pod \"payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:18.766169 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.766181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37dbd87-a283-40be-8729-f2ab95c1fa18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:18.766396 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.766272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e9dad6-5e65-4390-b518-1188c7d045c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.766396 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.766305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrjr\" (UniqueName: \"kubernetes.io/projected/a9e9dad6-5e65-4390-b518-1188c7d045c4-kube-api-access-fqrjr\") pod \"payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.867488 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.867389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e9dad6-5e65-4390-b518-1188c7d045c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.867488 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.867432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrjr\" (UniqueName: \"kubernetes.io/projected/a9e9dad6-5e65-4390-b518-1188c7d045c4-kube-api-access-fqrjr\") pod \"payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.867770 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.867523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbcd6\" (UniqueName: \"kubernetes.io/projected/b37dbd87-a283-40be-8729-f2ab95c1fa18-kube-api-access-rbcd6\") pod \"payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:18.867770 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.867694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37dbd87-a283-40be-8729-f2ab95c1fa18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:18.868112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.868093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e9dad6-5e65-4390-b518-1188c7d045c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.868373 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.868357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37dbd87-a283-40be-8729-f2ab95c1fa18-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:18.877016 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.876993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbcd6\" (UniqueName: \"kubernetes.io/projected/b37dbd87-a283-40be-8729-f2ab95c1fa18-kube-api-access-rbcd6\") pod \"payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:18.877016 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.877009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrjr\" (UniqueName: \"kubernetes.io/projected/a9e9dad6-5e65-4390-b518-1188c7d045c4-kube-api-access-fqrjr\") pod \"payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.929122 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.929076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:18.983374 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:18.983144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:19.208571 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:19.208530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"]
Sep 11 04:21:19.209842 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:21:19.209818    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e9dad6_5e65_4390_b518_1188c7d045c4.slice/crio-a19697a20dbd333062b785746bb856e7925641996a339e073befde5a141498ed WatchSource:0}: Error finding container a19697a20dbd333062b785746bb856e7925641996a339e073befde5a141498ed: Status 404 returned error can't find the container with id a19697a20dbd333062b785746bb856e7925641996a339e073befde5a141498ed
Sep 11 04:21:19.277227 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:19.277197    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"]
Sep 11 04:21:19.278282 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:21:19.278258    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37dbd87_a283_40be_8729_f2ab95c1fa18.slice/crio-e3487170815b5a2928bf1a7edf34673b4e375fdd8bf145339eec5baa031ff94a WatchSource:0}: Error finding container e3487170815b5a2928bf1a7edf34673b4e375fdd8bf145339eec5baa031ff94a: Status 404 returned error can't find the container with id e3487170815b5a2928bf1a7edf34673b4e375fdd8bf145339eec5baa031ff94a
Sep 11 04:21:20.186288 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.186248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7" event={"ID":"b37dbd87-a283-40be-8729-f2ab95c1fa18","Type":"ContainerStarted","Data":"8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544"}
Sep 11 04:21:20.186846 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.186313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7" event={"ID":"b37dbd87-a283-40be-8729-f2ab95c1fa18","Type":"ContainerStarted","Data":"e3487170815b5a2928bf1a7edf34673b4e375fdd8bf145339eec5baa031ff94a"}
Sep 11 04:21:20.186846 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.186371    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:20.188097 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.188061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j" event={"ID":"a9e9dad6-5e65-4390-b518-1188c7d045c4","Type":"ContainerStarted","Data":"35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f"}
Sep 11 04:21:20.188227 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.188117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j" event={"ID":"a9e9dad6-5e65-4390-b518-1188c7d045c4","Type":"ContainerStarted","Data":"a19697a20dbd333062b785746bb856e7925641996a339e073befde5a141498ed"}
Sep 11 04:21:20.188227 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.188149    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:20.206690 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.206633    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7" podStartSLOduration=2.20661264 podStartE2EDuration="2.20661264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:21:20.203632735 +0000 UTC m=+1654.468111109" watchObservedRunningTime="2026-09-11 04:21:20.20661264 +0000 UTC m=+1654.471091008"
Sep 11 04:21:20.223001 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:20.222946    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j" podStartSLOduration=2.222929729 podStartE2EDuration="2.222929729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:21:20.221250108 +0000 UTC m=+1654.485728479" watchObservedRunningTime="2026-09-11 04:21:20.222929729 +0000 UTC m=+1654.487408099"
Sep 11 04:21:31.195620 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:31.195583    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:31.196107 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:31.195845    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:42.637930 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.637886    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"]
Sep 11 04:21:42.643584 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.642821    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"
Sep 11 04:21:42.646734 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.646698    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:21:42.647637 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.647605    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:21:42.652657 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.652632    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"]
Sep 11 04:21:42.724078 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.724036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d039e8f8-636b-4612-9ef9-ea4f8569b26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9\" (UID: \"d039e8f8-636b-4612-9ef9-ea4f8569b26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"
Sep 11 04:21:42.824655 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.824609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d039e8f8-636b-4612-9ef9-ea4f8569b26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9\" (UID: \"d039e8f8-636b-4612-9ef9-ea4f8569b26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"
Sep 11 04:21:42.825260 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.825237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d039e8f8-636b-4612-9ef9-ea4f8569b26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9\" (UID: \"d039e8f8-636b-4612-9ef9-ea4f8569b26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"
Sep 11 04:21:42.958515 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:42.958464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"
Sep 11 04:21:43.231846 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:43.231798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"]
Sep 11 04:21:43.234289 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:21:43.234258    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd039e8f8_636b_4612_9ef9_ea4f8569b26a.slice/crio-51b9c505446c3b4220569e0fe6ee90ffa3081938e19cf55dec342a43a01b038d WatchSource:0}: Error finding container 51b9c505446c3b4220569e0fe6ee90ffa3081938e19cf55dec342a43a01b038d: Status 404 returned error can't find the container with id 51b9c505446c3b4220569e0fe6ee90ffa3081938e19cf55dec342a43a01b038d
Sep 11 04:21:43.319932 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:43.319893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9" event={"ID":"d039e8f8-636b-4612-9ef9-ea4f8569b26a","Type":"ContainerStarted","Data":"51b9c505446c3b4220569e0fe6ee90ffa3081938e19cf55dec342a43a01b038d"}
Sep 11 04:21:44.327251 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:44.327215    2582 generic.go:356] "Generic (PLEG): container finished" podID="d039e8f8-636b-4612-9ef9-ea4f8569b26a" containerID="61bff928d32b58ee52ebe27748a64abeae14ce8ceb4ef83906547eded0ee9ae4" exitCode=0
Sep 11 04:21:44.327720 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:44.327299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9" event={"ID":"d039e8f8-636b-4612-9ef9-ea4f8569b26a","Type":"ContainerDied","Data":"61bff928d32b58ee52ebe27748a64abeae14ce8ceb4ef83906547eded0ee9ae4"}
Sep 11 04:21:45.474336 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:45.474303    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"
Sep 11 04:21:45.553232 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:45.553174    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d039e8f8-636b-4612-9ef9-ea4f8569b26a-maas-api-service-ca\") pod \"d039e8f8-636b-4612-9ef9-ea4f8569b26a\" (UID: \"d039e8f8-636b-4612-9ef9-ea4f8569b26a\") "
Sep 11 04:21:45.553539 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:45.553515    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d039e8f8-636b-4612-9ef9-ea4f8569b26a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d039e8f8-636b-4612-9ef9-ea4f8569b26a" (UID: "d039e8f8-636b-4612-9ef9-ea4f8569b26a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:21:45.654437 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:45.654348    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d039e8f8-636b-4612-9ef9-ea4f8569b26a-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:21:46.340993 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:46.340960    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"
Sep 11 04:21:46.340993 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:46.340980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9" event={"ID":"d039e8f8-636b-4612-9ef9-ea4f8569b26a","Type":"ContainerDied","Data":"51b9c505446c3b4220569e0fe6ee90ffa3081938e19cf55dec342a43a01b038d"}
Sep 11 04:21:46.341196 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:46.341024    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51b9c505446c3b4220569e0fe6ee90ffa3081938e19cf55dec342a43a01b038d"
Sep 11 04:21:46.594577 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:46.594476    2582 scope.go:117] "RemoveContainer" containerID="ad330c1d3138ed2ce2320042a2b304c1acb5d65d29aa42401732623f3fe2ccfa"
Sep 11 04:21:46.603354 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:46.603331    2582 scope.go:117] "RemoveContainer" containerID="c400f801c1d30d3497404a45eee7b27143de611611ea4aabffcc4372654bbe04"
Sep 11 04:21:48.543948 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.543913    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"]
Sep 11 04:21:48.544515 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.544184    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j" podUID="a9e9dad6-5e65-4390-b518-1188c7d045c4" containerName="payload-processing" containerID="cri-o://35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f" gracePeriod=30
Sep 11 04:21:48.553972 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.553941    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"]
Sep 11 04:21:48.554228 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.554177    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7" podUID="b37dbd87-a283-40be-8729-f2ab95c1fa18" containerName="payload-pre-processing" containerID="cri-o://8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544" gracePeriod=30
Sep 11 04:21:48.708086 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.708046    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"]
Sep 11 04:21:48.711955 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.711925    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-84b423-84vc9"]
Sep 11 04:21:48.828296 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.828266    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:48.831580 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.831543    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:48.988707 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.988673    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e9dad6-5e65-4390-b518-1188c7d045c4-plugins-config-volume\") pod \"a9e9dad6-5e65-4390-b518-1188c7d045c4\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") "
Sep 11 04:21:48.988919 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.988734    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqrjr\" (UniqueName: \"kubernetes.io/projected/a9e9dad6-5e65-4390-b518-1188c7d045c4-kube-api-access-fqrjr\") pod \"a9e9dad6-5e65-4390-b518-1188c7d045c4\" (UID: \"a9e9dad6-5e65-4390-b518-1188c7d045c4\") "
Sep 11 04:21:48.988919 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.988821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbcd6\" (UniqueName: \"kubernetes.io/projected/b37dbd87-a283-40be-8729-f2ab95c1fa18-kube-api-access-rbcd6\") pod \"b37dbd87-a283-40be-8729-f2ab95c1fa18\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") "
Sep 11 04:21:48.988919 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.988838    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37dbd87-a283-40be-8729-f2ab95c1fa18-plugins-config-volume\") pod \"b37dbd87-a283-40be-8729-f2ab95c1fa18\" (UID: \"b37dbd87-a283-40be-8729-f2ab95c1fa18\") "
Sep 11 04:21:48.989176 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.989150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e9dad6-5e65-4390-b518-1188c7d045c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9e9dad6-5e65-4390-b518-1188c7d045c4" (UID: "a9e9dad6-5e65-4390-b518-1188c7d045c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:21:48.989302 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.989281    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37dbd87-a283-40be-8729-f2ab95c1fa18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b37dbd87-a283-40be-8729-f2ab95c1fa18" (UID: "b37dbd87-a283-40be-8729-f2ab95c1fa18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:21:48.990964 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.990939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e9dad6-5e65-4390-b518-1188c7d045c4-kube-api-access-fqrjr" (OuterVolumeSpecName: "kube-api-access-fqrjr") pod "a9e9dad6-5e65-4390-b518-1188c7d045c4" (UID: "a9e9dad6-5e65-4390-b518-1188c7d045c4"). InnerVolumeSpecName "kube-api-access-fqrjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:21:48.991030 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:48.990991    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37dbd87-a283-40be-8729-f2ab95c1fa18-kube-api-access-rbcd6" (OuterVolumeSpecName: "kube-api-access-rbcd6") pod "b37dbd87-a283-40be-8729-f2ab95c1fa18" (UID: "b37dbd87-a283-40be-8729-f2ab95c1fa18"). InnerVolumeSpecName "kube-api-access-rbcd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:21:49.090355 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.090263    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbcd6\" (UniqueName: \"kubernetes.io/projected/b37dbd87-a283-40be-8729-f2ab95c1fa18-kube-api-access-rbcd6\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:21:49.090355 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.090296    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37dbd87-a283-40be-8729-f2ab95c1fa18-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:21:49.090355 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.090305    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e9dad6-5e65-4390-b518-1188c7d045c4-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:21:49.090355 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.090314    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqrjr\" (UniqueName: \"kubernetes.io/projected/a9e9dad6-5e65-4390-b518-1188c7d045c4-kube-api-access-fqrjr\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:21:49.366679 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.366591    2582 generic.go:356] "Generic (PLEG): container finished" podID="a9e9dad6-5e65-4390-b518-1188c7d045c4" containerID="35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f" exitCode=0
Sep 11 04:21:49.366836 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.366685    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"
Sep 11 04:21:49.366836 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.366722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j" event={"ID":"a9e9dad6-5e65-4390-b518-1188c7d045c4","Type":"ContainerDied","Data":"35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f"}
Sep 11 04:21:49.366836 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.366776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j" event={"ID":"a9e9dad6-5e65-4390-b518-1188c7d045c4","Type":"ContainerDied","Data":"a19697a20dbd333062b785746bb856e7925641996a339e073befde5a141498ed"}
Sep 11 04:21:49.366836 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.366804    2582 scope.go:117] "RemoveContainer" containerID="35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f"
Sep 11 04:21:49.368267 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.368246    2582 generic.go:356] "Generic (PLEG): container finished" podID="b37dbd87-a283-40be-8729-f2ab95c1fa18" containerID="8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544" exitCode=0
Sep 11 04:21:49.368382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.368281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7" event={"ID":"b37dbd87-a283-40be-8729-f2ab95c1fa18","Type":"ContainerDied","Data":"8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544"}
Sep 11 04:21:49.368382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.368295    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"
Sep 11 04:21:49.368382 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.368317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7" event={"ID":"b37dbd87-a283-40be-8729-f2ab95c1fa18","Type":"ContainerDied","Data":"e3487170815b5a2928bf1a7edf34673b4e375fdd8bf145339eec5baa031ff94a"}
Sep 11 04:21:49.379014 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.378993    2582 scope.go:117] "RemoveContainer" containerID="35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f"
Sep 11 04:21:49.379363 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:21:49.379333    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f\": container with ID starting with 35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f not found: ID does not exist" containerID="35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f"
Sep 11 04:21:49.379454 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.379374    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f"} err="failed to get container status \"35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f\": rpc error: code = NotFound desc = could not find container \"35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f\": container with ID starting with 35b109a3625e47c60a0bbffc480d2e9b2108d9410b9f2ee7fb9186a46a51409f not found: ID does not exist"
Sep 11 04:21:49.379454 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.379398    2582 scope.go:117] "RemoveContainer" containerID="8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544"
Sep 11 04:21:49.389113 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.389092    2582 scope.go:117] "RemoveContainer" containerID="8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544"
Sep 11 04:21:49.389434 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:21:49.389416    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544\": container with ID starting with 8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544 not found: ID does not exist" containerID="8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544"
Sep 11 04:21:49.389501 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.389443    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544"} err="failed to get container status \"8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544\": rpc error: code = NotFound desc = could not find container \"8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544\": container with ID starting with 8ec067d6cdbc4c78ef37b1f2e772541e2c8b5bb5e8721230090ef9952a3d6544 not found: ID does not exist"
Sep 11 04:21:49.390023 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.390004    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"]
Sep 11 04:21:49.395747 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.395724    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-84b423-7f58594789-8pr5j"]
Sep 11 04:21:49.407421 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.407395    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"]
Sep 11 04:21:49.412077 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:49.412053    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-84b423-6dfdfcb5c8-xw5j7"]
Sep 11 04:21:50.274937 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:50.274904    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e9dad6-5e65-4390-b518-1188c7d045c4" path="/var/lib/kubelet/pods/a9e9dad6-5e65-4390-b518-1188c7d045c4/volumes"
Sep 11 04:21:50.275306 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:50.275256    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37dbd87-a283-40be-8729-f2ab95c1fa18" path="/var/lib/kubelet/pods/b37dbd87-a283-40be-8729-f2ab95c1fa18/volumes"
Sep 11 04:21:50.275548 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:50.275535    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d039e8f8-636b-4612-9ef9-ea4f8569b26a" path="/var/lib/kubelet/pods/d039e8f8-636b-4612-9ef9-ea4f8569b26a/volumes"
Sep 11 04:21:59.165086 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165048    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"]
Sep 11 04:21:59.165516 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165502    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e9dad6-5e65-4390-b518-1188c7d045c4" containerName="payload-processing"
Sep 11 04:21:59.165578 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165518    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e9dad6-5e65-4390-b518-1188c7d045c4" containerName="payload-processing"
Sep 11 04:21:59.165578 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165538    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d039e8f8-636b-4612-9ef9-ea4f8569b26a" containerName="revoke-keys"
Sep 11 04:21:59.165578 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165543    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d039e8f8-636b-4612-9ef9-ea4f8569b26a" containerName="revoke-keys"
Sep 11 04:21:59.165578 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165570    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37dbd87-a283-40be-8729-f2ab95c1fa18" containerName="payload-pre-processing"
Sep 11 04:21:59.165578 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165576    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37dbd87-a283-40be-8729-f2ab95c1fa18" containerName="payload-pre-processing"
Sep 11 04:21:59.165745 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165648    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d039e8f8-636b-4612-9ef9-ea4f8569b26a" containerName="revoke-keys"
Sep 11 04:21:59.165745 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165656    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e9dad6-5e65-4390-b518-1188c7d045c4" containerName="payload-processing"
Sep 11 04:21:59.165745 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.165665    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37dbd87-a283-40be-8729-f2ab95c1fa18" containerName="payload-pre-processing"
Sep 11 04:21:59.170387 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.170369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"
Sep 11 04:21:59.174220 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.174193    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:21:59.174579 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.174266    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:21:59.176455 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.176429    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"]
Sep 11 04:21:59.179524 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.179496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abff231-93b6-41c2-b4b9-d4c26381a82f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w\" (UID: \"4abff231-93b6-41c2-b4b9-d4c26381a82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"
Sep 11 04:21:59.280131 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.280072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abff231-93b6-41c2-b4b9-d4c26381a82f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w\" (UID: \"4abff231-93b6-41c2-b4b9-d4c26381a82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"
Sep 11 04:21:59.280804 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.280780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abff231-93b6-41c2-b4b9-d4c26381a82f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w\" (UID: \"4abff231-93b6-41c2-b4b9-d4c26381a82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"
Sep 11 04:21:59.482881 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.482845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"
Sep 11 04:21:59.756676 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:21:59.756649    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"]
Sep 11 04:21:59.759147 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:21:59.759116    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abff231_93b6_41c2_b4b9_d4c26381a82f.slice/crio-147eb5046672843e5d75012aa02ced358d7349db7933bc7d1eb9594d97c10f47 WatchSource:0}: Error finding container 147eb5046672843e5d75012aa02ced358d7349db7933bc7d1eb9594d97c10f47: Status 404 returned error can't find the container with id 147eb5046672843e5d75012aa02ced358d7349db7933bc7d1eb9594d97c10f47
Sep 11 04:22:00.435396 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:00.435363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w" event={"ID":"4abff231-93b6-41c2-b4b9-d4c26381a82f","Type":"ContainerStarted","Data":"451b640c8c7e5e22b4c5f1341ea3ee050d7bb83130679d76a34189ebe43433c4"}
Sep 11 04:22:00.435776 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:00.435419    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w" event={"ID":"4abff231-93b6-41c2-b4b9-d4c26381a82f","Type":"ContainerStarted","Data":"147eb5046672843e5d75012aa02ced358d7349db7933bc7d1eb9594d97c10f47"}
Sep 11 04:22:00.456250 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:00.456197    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w" podStartSLOduration=1.456179624 podStartE2EDuration="1.456179624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:22:00.453795731 +0000 UTC m=+1694.718274101" watchObservedRunningTime="2026-09-11 04:22:00.456179624 +0000 UTC m=+1694.720657995"
Sep 11 04:22:01.443304 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:01.443264    2582 generic.go:356] "Generic (PLEG): container finished" podID="4abff231-93b6-41c2-b4b9-d4c26381a82f" containerID="451b640c8c7e5e22b4c5f1341ea3ee050d7bb83130679d76a34189ebe43433c4" exitCode=0
Sep 11 04:22:01.443769 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:01.443311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w" event={"ID":"4abff231-93b6-41c2-b4b9-d4c26381a82f","Type":"ContainerDied","Data":"451b640c8c7e5e22b4c5f1341ea3ee050d7bb83130679d76a34189ebe43433c4"}
Sep 11 04:22:02.581444 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:02.581420    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"
Sep 11 04:22:02.609102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:02.609024    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abff231-93b6-41c2-b4b9-d4c26381a82f-maas-api-service-ca\") pod \"4abff231-93b6-41c2-b4b9-d4c26381a82f\" (UID: \"4abff231-93b6-41c2-b4b9-d4c26381a82f\") "
Sep 11 04:22:02.609355 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:02.609332    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abff231-93b6-41c2-b4b9-d4c26381a82f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4abff231-93b6-41c2-b4b9-d4c26381a82f" (UID: "4abff231-93b6-41c2-b4b9-d4c26381a82f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:22:02.710340 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:02.710246    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abff231-93b6-41c2-b4b9-d4c26381a82f-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:22:03.457799 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:03.457769    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"
Sep 11 04:22:03.457990 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:03.457770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w" event={"ID":"4abff231-93b6-41c2-b4b9-d4c26381a82f","Type":"ContainerDied","Data":"147eb5046672843e5d75012aa02ced358d7349db7933bc7d1eb9594d97c10f47"}
Sep 11 04:22:03.457990 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:03.457889    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147eb5046672843e5d75012aa02ced358d7349db7933bc7d1eb9594d97c10f47"
Sep 11 04:22:14.810596 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:14.810498    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"]
Sep 11 04:22:14.811010 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:14.810766    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4" podUID="d4cb6617-fe7e-4297-98c0-97c7668cd1fc" containerName="payload-processing" containerID="cri-o://a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef" gracePeriod=30
Sep 11 04:22:14.820368 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:14.820341    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"]
Sep 11 04:22:14.820736 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:14.820709    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs" podUID="fb936454-d315-4b34-a7a2-857ab57efb75" containerName="payload-pre-processing" containerID="cri-o://3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16" gracePeriod=30
Sep 11 04:22:15.035940 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.035849    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"]
Sep 11 04:22:15.039387 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.039324    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fdc690-g6b7w"]
Sep 11 04:22:15.067234 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.067172    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:22:15.097888 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.097853    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:22:15.134835 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.134799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-plugins-config-volume\") pod \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") "
Sep 11 04:22:15.135035 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.134981    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqzgj\" (UniqueName: \"kubernetes.io/projected/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-kube-api-access-jqzgj\") pod \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\" (UID: \"d4cb6617-fe7e-4297-98c0-97c7668cd1fc\") "
Sep 11 04:22:15.135232 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.135207    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4cb6617-fe7e-4297-98c0-97c7668cd1fc" (UID: "d4cb6617-fe7e-4297-98c0-97c7668cd1fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:22:15.140603 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.135855    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:22:15.140603 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.137735    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-kube-api-access-jqzgj" (OuterVolumeSpecName: "kube-api-access-jqzgj") pod "d4cb6617-fe7e-4297-98c0-97c7668cd1fc" (UID: "d4cb6617-fe7e-4297-98c0-97c7668cd1fc"). InnerVolumeSpecName "kube-api-access-jqzgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:22:15.236482 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.236440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s9qn\" (UniqueName: \"kubernetes.io/projected/fb936454-d315-4b34-a7a2-857ab57efb75-kube-api-access-5s9qn\") pod \"fb936454-d315-4b34-a7a2-857ab57efb75\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") "
Sep 11 04:22:15.236688 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.236508    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb936454-d315-4b34-a7a2-857ab57efb75-plugins-config-volume\") pod \"fb936454-d315-4b34-a7a2-857ab57efb75\" (UID: \"fb936454-d315-4b34-a7a2-857ab57efb75\") "
Sep 11 04:22:15.236798 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.236784    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqzgj\" (UniqueName: \"kubernetes.io/projected/d4cb6617-fe7e-4297-98c0-97c7668cd1fc-kube-api-access-jqzgj\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:22:15.236917 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.236884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb936454-d315-4b34-a7a2-857ab57efb75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb936454-d315-4b34-a7a2-857ab57efb75" (UID: "fb936454-d315-4b34-a7a2-857ab57efb75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:22:15.238605 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.238580    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb936454-d315-4b34-a7a2-857ab57efb75-kube-api-access-5s9qn" (OuterVolumeSpecName: "kube-api-access-5s9qn") pod "fb936454-d315-4b34-a7a2-857ab57efb75" (UID: "fb936454-d315-4b34-a7a2-857ab57efb75"). InnerVolumeSpecName "kube-api-access-5s9qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:22:15.338297 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.338189    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s9qn\" (UniqueName: \"kubernetes.io/projected/fb936454-d315-4b34-a7a2-857ab57efb75-kube-api-access-5s9qn\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:22:15.338297 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.338233    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb936454-d315-4b34-a7a2-857ab57efb75-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:22:15.523525 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.523485    2582 generic.go:356] "Generic (PLEG): container finished" podID="d4cb6617-fe7e-4297-98c0-97c7668cd1fc" containerID="a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef" exitCode=0
Sep 11 04:22:15.523731 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.523538    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"
Sep 11 04:22:15.523731 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.523586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4" event={"ID":"d4cb6617-fe7e-4297-98c0-97c7668cd1fc","Type":"ContainerDied","Data":"a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef"}
Sep 11 04:22:15.523731 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.523645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4" event={"ID":"d4cb6617-fe7e-4297-98c0-97c7668cd1fc","Type":"ContainerDied","Data":"ee9e38755d4f8d1aaca8ac6dbfcd771544ebb6a42bd4bbbff6b59070d6e89da3"}
Sep 11 04:22:15.523731 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.523674    2582 scope.go:117] "RemoveContainer" containerID="a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef"
Sep 11 04:22:15.525007 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.524984    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb936454-d315-4b34-a7a2-857ab57efb75" containerID="3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16" exitCode=0
Sep 11 04:22:15.525109 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.525021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs" event={"ID":"fb936454-d315-4b34-a7a2-857ab57efb75","Type":"ContainerDied","Data":"3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16"}
Sep 11 04:22:15.525109 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.525057    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"
Sep 11 04:22:15.525109 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.525071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs" event={"ID":"fb936454-d315-4b34-a7a2-857ab57efb75","Type":"ContainerDied","Data":"4b64f44e33f680f91ee2e420c937ae7d4ff41a4fe698764d7931088c9478dfed"}
Sep 11 04:22:15.533117 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.533093    2582 scope.go:117] "RemoveContainer" containerID="a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef"
Sep 11 04:22:15.533488 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:22:15.533447    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef\": container with ID starting with a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef not found: ID does not exist" containerID="a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef"
Sep 11 04:22:15.533635 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.533491    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef"} err="failed to get container status \"a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef\": rpc error: code = NotFound desc = could not find container \"a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef\": container with ID starting with a17949aaa6e0caeabc28b477cf7ba63d7e249d84775dbc1a1a9f5304a3cd82ef not found: ID does not exist"
Sep 11 04:22:15.533635 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.533515    2582 scope.go:117] "RemoveContainer" containerID="3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16"
Sep 11 04:22:15.542906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.542884    2582 scope.go:117] "RemoveContainer" containerID="3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16"
Sep 11 04:22:15.543188 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:22:15.543169    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16\": container with ID starting with 3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16 not found: ID does not exist" containerID="3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16"
Sep 11 04:22:15.543234 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.543196    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16"} err="failed to get container status \"3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16\": rpc error: code = NotFound desc = could not find container \"3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16\": container with ID starting with 3f00646bc5c1ef4f5a93718d7460ba8358864e07de526aac18197ed747406e16 not found: ID does not exist"
Sep 11 04:22:15.551101 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.551078    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"]
Sep 11 04:22:15.555199 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.555172    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fdc690-6fcc6c49cc-nr6c4"]
Sep 11 04:22:15.567445 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.567416    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"]
Sep 11 04:22:15.573598 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:15.573540    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fdc690-848fb6554b-8lbfs"]
Sep 11 04:22:16.276417 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:16.276384    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4abff231-93b6-41c2-b4b9-d4c26381a82f" path="/var/lib/kubelet/pods/4abff231-93b6-41c2-b4b9-d4c26381a82f/volumes"
Sep 11 04:22:16.276839 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:16.276823    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4cb6617-fe7e-4297-98c0-97c7668cd1fc" path="/var/lib/kubelet/pods/d4cb6617-fe7e-4297-98c0-97c7668cd1fc/volumes"
Sep 11 04:22:16.277152 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:22:16.277141    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb936454-d315-4b34-a7a2-857ab57efb75" path="/var/lib/kubelet/pods/fb936454-d315-4b34-a7a2-857ab57efb75/volumes"
Sep 11 04:23:04.035153 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035108    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"]
Sep 11 04:23:04.035691 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035632    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4cb6617-fe7e-4297-98c0-97c7668cd1fc" containerName="payload-processing"
Sep 11 04:23:04.035691 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035650    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4cb6617-fe7e-4297-98c0-97c7668cd1fc" containerName="payload-processing"
Sep 11 04:23:04.035691 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035673    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abff231-93b6-41c2-b4b9-d4c26381a82f" containerName="revoke-keys"
Sep 11 04:23:04.035691 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035679    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abff231-93b6-41c2-b4b9-d4c26381a82f" containerName="revoke-keys"
Sep 11 04:23:04.035691 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035688    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb936454-d315-4b34-a7a2-857ab57efb75" containerName="payload-pre-processing"
Sep 11 04:23:04.035691 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035694    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb936454-d315-4b34-a7a2-857ab57efb75" containerName="payload-pre-processing"
Sep 11 04:23:04.035901 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035759    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4abff231-93b6-41c2-b4b9-d4c26381a82f" containerName="revoke-keys"
Sep 11 04:23:04.035901 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035767    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb936454-d315-4b34-a7a2-857ab57efb75" containerName="payload-pre-processing"
Sep 11 04:23:04.035901 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.035775    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4cb6617-fe7e-4297-98c0-97c7668cd1fc" containerName="payload-processing"
Sep 11 04:23:04.038723 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.038705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"
Sep 11 04:23:04.041349 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.041323    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:23:04.041455 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.041338    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:23:04.045700 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.045671    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"]
Sep 11 04:23:04.128717 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.128681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf39587-b196-40ad-a171-64df48ed9387-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz\" (UID: \"4cf39587-b196-40ad-a171-64df48ed9387\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"
Sep 11 04:23:04.229348 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.229309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf39587-b196-40ad-a171-64df48ed9387-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz\" (UID: \"4cf39587-b196-40ad-a171-64df48ed9387\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"
Sep 11 04:23:04.230038 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.230012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf39587-b196-40ad-a171-64df48ed9387-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz\" (UID: \"4cf39587-b196-40ad-a171-64df48ed9387\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"
Sep 11 04:23:04.350502 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.350413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"
Sep 11 04:23:04.622259 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.622234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"]
Sep 11 04:23:04.624320 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:23:04.624285    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf39587_b196_40ad_a171_64df48ed9387.slice/crio-e819d9bed362995592b72b311322b743e8f75ca1234b5b0987f4aa0d16a06473 WatchSource:0}: Error finding container e819d9bed362995592b72b311322b743e8f75ca1234b5b0987f4aa0d16a06473: Status 404 returned error can't find the container with id e819d9bed362995592b72b311322b743e8f75ca1234b5b0987f4aa0d16a06473
Sep 11 04:23:04.811749 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:04.811717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz" event={"ID":"4cf39587-b196-40ad-a171-64df48ed9387","Type":"ContainerStarted","Data":"e819d9bed362995592b72b311322b743e8f75ca1234b5b0987f4aa0d16a06473"}
Sep 11 04:23:05.819575 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:05.819519    2582 generic.go:356] "Generic (PLEG): container finished" podID="4cf39587-b196-40ad-a171-64df48ed9387" containerID="60a3875f8c63685ceefc00d74b18c012b7cd2ba5073d8fff459f8a15ba39a437" exitCode=0
Sep 11 04:23:05.820032 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:05.819613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz" event={"ID":"4cf39587-b196-40ad-a171-64df48ed9387","Type":"ContainerDied","Data":"60a3875f8c63685ceefc00d74b18c012b7cd2ba5073d8fff459f8a15ba39a437"}
Sep 11 04:23:06.960632 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:06.960606    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"
Sep 11 04:23:07.062077 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:07.062035    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf39587-b196-40ad-a171-64df48ed9387-maas-api-service-ca\") pod \"4cf39587-b196-40ad-a171-64df48ed9387\" (UID: \"4cf39587-b196-40ad-a171-64df48ed9387\") "
Sep 11 04:23:07.062415 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:07.062381    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf39587-b196-40ad-a171-64df48ed9387-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cf39587-b196-40ad-a171-64df48ed9387" (UID: "4cf39587-b196-40ad-a171-64df48ed9387"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:07.162891 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:07.162806    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf39587-b196-40ad-a171-64df48ed9387-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:07.832742 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:07.832714    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"
Sep 11 04:23:07.832742 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:07.832729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz" event={"ID":"4cf39587-b196-40ad-a171-64df48ed9387","Type":"ContainerDied","Data":"e819d9bed362995592b72b311322b743e8f75ca1234b5b0987f4aa0d16a06473"}
Sep 11 04:23:07.832966 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:07.832773    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e819d9bed362995592b72b311322b743e8f75ca1234b5b0987f4aa0d16a06473"
Sep 11 04:23:14.621379 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.621337    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"]
Sep 11 04:23:14.621894 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.621570    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx" podUID="980aeac0-f71d-4004-be26-9e9125947738" containerName="payload-pre-processing" containerID="cri-o://b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c" gracePeriod=30
Sep 11 04:23:14.781079 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.781017    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"]
Sep 11 04:23:14.783442 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.783414    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a5a85-l6ffz"]
Sep 11 04:23:14.865943 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.865916    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:23:14.875372 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.875300    2582 generic.go:356] "Generic (PLEG): container finished" podID="980aeac0-f71d-4004-be26-9e9125947738" containerID="b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c" exitCode=0
Sep 11 04:23:14.875372 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.875336    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"
Sep 11 04:23:14.875372 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.875347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx" event={"ID":"980aeac0-f71d-4004-be26-9e9125947738","Type":"ContainerDied","Data":"b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c"}
Sep 11 04:23:14.875619 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.875400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx" event={"ID":"980aeac0-f71d-4004-be26-9e9125947738","Type":"ContainerDied","Data":"0187880b960c9c06fbf291c5967202c0434527bc910f4e53e8820693e6f17729"}
Sep 11 04:23:14.875619 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.875423    2582 scope.go:117] "RemoveContainer" containerID="b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c"
Sep 11 04:23:14.885366 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.885275    2582 scope.go:117] "RemoveContainer" containerID="b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c"
Sep 11 04:23:14.885753 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:23:14.885728    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c\": container with ID starting with b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c not found: ID does not exist" containerID="b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c"
Sep 11 04:23:14.885847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.885764    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c"} err="failed to get container status \"b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c\": rpc error: code = NotFound desc = could not find container \"b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c\": container with ID starting with b7018d37007b9f333a06fd671d7b6c58e9ec0da6c29f669809a31b498088589c not found: ID does not exist"
Sep 11 04:23:14.938304 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.938269    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvgwz\" (UniqueName: \"kubernetes.io/projected/980aeac0-f71d-4004-be26-9e9125947738-kube-api-access-kvgwz\") pod \"980aeac0-f71d-4004-be26-9e9125947738\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") "
Sep 11 04:23:14.938475 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.938315    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980aeac0-f71d-4004-be26-9e9125947738-plugins-config-volume\") pod \"980aeac0-f71d-4004-be26-9e9125947738\" (UID: \"980aeac0-f71d-4004-be26-9e9125947738\") "
Sep 11 04:23:14.938752 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.938718    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980aeac0-f71d-4004-be26-9e9125947738-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "980aeac0-f71d-4004-be26-9e9125947738" (UID: "980aeac0-f71d-4004-be26-9e9125947738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:14.940434 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:14.940410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980aeac0-f71d-4004-be26-9e9125947738-kube-api-access-kvgwz" (OuterVolumeSpecName: "kube-api-access-kvgwz") pod "980aeac0-f71d-4004-be26-9e9125947738" (UID: "980aeac0-f71d-4004-be26-9e9125947738"). InnerVolumeSpecName "kube-api-access-kvgwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:15.039989 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:15.039949    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvgwz\" (UniqueName: \"kubernetes.io/projected/980aeac0-f71d-4004-be26-9e9125947738-kube-api-access-kvgwz\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:15.039989 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:15.039983    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/980aeac0-f71d-4004-be26-9e9125947738-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:15.198056 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:15.198018    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"]
Sep 11 04:23:15.208308 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:15.208276    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a5a85-699b88f8bb-dzrsx"]
Sep 11 04:23:16.274919 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:16.274883    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf39587-b196-40ad-a171-64df48ed9387" path="/var/lib/kubelet/pods/4cf39587-b196-40ad-a171-64df48ed9387/volumes"
Sep 11 04:23:16.275295 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:16.275249    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980aeac0-f71d-4004-be26-9e9125947738" path="/var/lib/kubelet/pods/980aeac0-f71d-4004-be26-9e9125947738/volumes"
Sep 11 04:23:21.560297 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.560258    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"]
Sep 11 04:23:21.560847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.560706    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980aeac0-f71d-4004-be26-9e9125947738" containerName="payload-pre-processing"
Sep 11 04:23:21.560847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.560719    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="980aeac0-f71d-4004-be26-9e9125947738" containerName="payload-pre-processing"
Sep 11 04:23:21.560847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.560741    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf39587-b196-40ad-a171-64df48ed9387" containerName="revoke-keys"
Sep 11 04:23:21.560847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.560750    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf39587-b196-40ad-a171-64df48ed9387" containerName="revoke-keys"
Sep 11 04:23:21.560847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.560836    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf39587-b196-40ad-a171-64df48ed9387" containerName="revoke-keys"
Sep 11 04:23:21.560847 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.560841    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="980aeac0-f71d-4004-be26-9e9125947738" containerName="payload-pre-processing"
Sep 11 04:23:21.563882 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.563862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"
Sep 11 04:23:21.566472 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.566451    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 04:23:21.566591 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.566453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:23:21.570875 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.570846    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"]
Sep 11 04:23:21.603255 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.603215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj\" (UID: \"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"
Sep 11 04:23:21.703958 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.703916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj\" (UID: \"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"
Sep 11 04:23:21.704619 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.704597    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj\" (UID: \"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"
Sep 11 04:23:21.876238 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:21.876150    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"
Sep 11 04:23:22.152809 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:22.152643    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"]
Sep 11 04:23:22.155852 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:23:22.155819    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb4ce6a_0ff2_4df0_81ba_d76ea8ee0d3b.slice/crio-47321fc32f3270f1c3c049b61bad681acab44d5bdf21f5698add76bc8231a425 WatchSource:0}: Error finding container 47321fc32f3270f1c3c049b61bad681acab44d5bdf21f5698add76bc8231a425: Status 404 returned error can't find the container with id 47321fc32f3270f1c3c049b61bad681acab44d5bdf21f5698add76bc8231a425
Sep 11 04:23:22.923647 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:22.923613    2582 generic.go:356] "Generic (PLEG): container finished" podID="2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b" containerID="6322611297b6a8472053b2e15aaff7435e34abaa71462eba532a3abd5d859520" exitCode=0
Sep 11 04:23:22.923985 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:22.923660    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj" event={"ID":"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b","Type":"ContainerDied","Data":"6322611297b6a8472053b2e15aaff7435e34abaa71462eba532a3abd5d859520"}
Sep 11 04:23:22.923985 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:22.923730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj" event={"ID":"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b","Type":"ContainerStarted","Data":"47321fc32f3270f1c3c049b61bad681acab44d5bdf21f5698add76bc8231a425"}
Sep 11 04:23:24.069944 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:24.069915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"
Sep 11 04:23:24.127662 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:24.127628    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b-maas-api-service-ca\") pod \"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b\" (UID: \"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b\") "
Sep 11 04:23:24.128060 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:24.128037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b" (UID: "2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:24.229176 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:24.229101    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b-maas-api-service-ca\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:24.937851 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:24.937821    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"
Sep 11 04:23:24.938030 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:24.937820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj" event={"ID":"2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b","Type":"ContainerDied","Data":"47321fc32f3270f1c3c049b61bad681acab44d5bdf21f5698add76bc8231a425"}
Sep 11 04:23:24.938030 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:24.937948    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47321fc32f3270f1c3c049b61bad681acab44d5bdf21f5698add76bc8231a425"
Sep 11 04:23:32.044258 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.044222    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"]
Sep 11 04:23:32.047885 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.047843    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cc6708-29818335-zz485"]
Sep 11 04:23:32.076050 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.076013    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"]
Sep 11 04:23:32.076335 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.076307    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6" podUID="4405beba-8ea0-47c8-a63e-2157673771b0" containerName="payload-processing" containerID="cri-o://73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060" gracePeriod=30
Sep 11 04:23:32.256446 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.256407    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"]
Sep 11 04:23:32.258384 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.258358    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cc6708-428jj"]
Sep 11 04:23:32.278311 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.278275    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b" path="/var/lib/kubelet/pods/2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b/volumes"
Sep 11 04:23:32.278747 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.278724    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df33797-19cc-4a11-b70c-06c12ab7def9" path="/var/lib/kubelet/pods/3df33797-19cc-4a11-b70c-06c12ab7def9/volumes"
Sep 11 04:23:32.339760 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.339729    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:23:32.408134 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.408092    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94z8x\" (UniqueName: \"kubernetes.io/projected/4405beba-8ea0-47c8-a63e-2157673771b0-kube-api-access-94z8x\") pod \"4405beba-8ea0-47c8-a63e-2157673771b0\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") "
Sep 11 04:23:32.408322 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.408144    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405beba-8ea0-47c8-a63e-2157673771b0-plugins-config-volume\") pod \"4405beba-8ea0-47c8-a63e-2157673771b0\" (UID: \"4405beba-8ea0-47c8-a63e-2157673771b0\") "
Sep 11 04:23:32.408739 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.408551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4405beba-8ea0-47c8-a63e-2157673771b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4405beba-8ea0-47c8-a63e-2157673771b0" (UID: "4405beba-8ea0-47c8-a63e-2157673771b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:32.409042 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.408975    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405beba-8ea0-47c8-a63e-2157673771b0-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:32.410607 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.410587    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4405beba-8ea0-47c8-a63e-2157673771b0-kube-api-access-94z8x" (OuterVolumeSpecName: "kube-api-access-94z8x") pod "4405beba-8ea0-47c8-a63e-2157673771b0" (UID: "4405beba-8ea0-47c8-a63e-2157673771b0"). InnerVolumeSpecName "kube-api-access-94z8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:32.509788 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.509750    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94z8x\" (UniqueName: \"kubernetes.io/projected/4405beba-8ea0-47c8-a63e-2157673771b0-kube-api-access-94z8x\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:32.985154 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.985117    2582 generic.go:356] "Generic (PLEG): container finished" podID="4405beba-8ea0-47c8-a63e-2157673771b0" containerID="73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060" exitCode=0
Sep 11 04:23:32.985321 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.985171    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"
Sep 11 04:23:32.985321 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.985198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6" event={"ID":"4405beba-8ea0-47c8-a63e-2157673771b0","Type":"ContainerDied","Data":"73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060"}
Sep 11 04:23:32.985321 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.985270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6" event={"ID":"4405beba-8ea0-47c8-a63e-2157673771b0","Type":"ContainerDied","Data":"c2e0a8bbd3ea02147bb311ea86797925ced9f85b845450344c3f2362c4e91b6e"}
Sep 11 04:23:32.985321 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.985295    2582 scope.go:117] "RemoveContainer" containerID="73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060"
Sep 11 04:23:32.994924 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.994903    2582 scope.go:117] "RemoveContainer" containerID="73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060"
Sep 11 04:23:32.995205 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:23:32.995183    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060\": container with ID starting with 73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060 not found: ID does not exist" containerID="73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060"
Sep 11 04:23:32.995276 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:32.995211    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060"} err="failed to get container status \"73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060\": rpc error: code = NotFound desc = could not find container \"73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060\": container with ID starting with 73164730ff492a08bf3429fdd22bf4f8bd651f0e3ed8836d449b56858f17f060 not found: ID does not exist"
Sep 11 04:23:33.008181 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:33.008153    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"]
Sep 11 04:23:33.012192 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:33.012171    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-cc6708-87cc5b7df-n9pb6"]
Sep 11 04:23:34.275665 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:34.275631    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4405beba-8ea0-47c8-a63e-2157673771b0" path="/var/lib/kubelet/pods/4405beba-8ea0-47c8-a63e-2157673771b0/volumes"
Sep 11 04:23:46.365971 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:46.365872    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:23:46.372730 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:46.370031    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:23:46.654918 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:46.654829    2582 scope.go:117] "RemoveContainer" containerID="54fbefb1813050183d91090b8bcb1c8aa90dc41a9ce135aa0e20775166f830ee"
Sep 11 04:23:49.606674 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.606633    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"]
Sep 11 04:23:49.608470 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.608447    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-9342ed-29818335-l7qh9"]
Sep 11 04:23:49.632029 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.631995    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"]
Sep 11 04:23:49.632266 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.632219    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz" podUID="975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" containerName="payload-processing" containerID="cri-o://d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042" gracePeriod=30
Sep 11 04:23:49.645634 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.645598    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"]
Sep 11 04:23:49.645866 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.645838    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww" podUID="3ed87f14-37c6-4178-ba0c-aafba2fbf63c" containerName="payload-pre-processing" containerID="cri-o://1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911" gracePeriod=30
Sep 11 04:23:49.915547 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.915520    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:23:49.918581 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.918546    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:23:49.987710 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.987669    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-plugins-config-volume\") pod \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") "
Sep 11 04:23:49.987906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.987723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg92l\" (UniqueName: \"kubernetes.io/projected/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-kube-api-access-xg92l\") pod \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\" (UID: \"3ed87f14-37c6-4178-ba0c-aafba2fbf63c\") "
Sep 11 04:23:49.987906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.987826    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-plugins-config-volume\") pod \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") "
Sep 11 04:23:49.987906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.987844    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xgkp\" (UniqueName: \"kubernetes.io/projected/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-kube-api-access-9xgkp\") pod \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\" (UID: \"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1\") "
Sep 11 04:23:49.988172 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.988054    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ed87f14-37c6-4178-ba0c-aafba2fbf63c" (UID: "3ed87f14-37c6-4178-ba0c-aafba2fbf63c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:49.988293 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.988251    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" (UID: "975c1895-7f6a-44bb-aaa3-03bb1a7c47f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:49.989993 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.989970    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-kube-api-access-9xgkp" (OuterVolumeSpecName: "kube-api-access-9xgkp") pod "975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" (UID: "975c1895-7f6a-44bb-aaa3-03bb1a7c47f1"). InnerVolumeSpecName "kube-api-access-9xgkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:49.990102 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:49.990088    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-kube-api-access-xg92l" (OuterVolumeSpecName: "kube-api-access-xg92l") pod "3ed87f14-37c6-4178-ba0c-aafba2fbf63c" (UID: "3ed87f14-37c6-4178-ba0c-aafba2fbf63c"). InnerVolumeSpecName "kube-api-access-xg92l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:50.088645 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.088615    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:50.088645 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.088643    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg92l\" (UniqueName: \"kubernetes.io/projected/3ed87f14-37c6-4178-ba0c-aafba2fbf63c-kube-api-access-xg92l\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:50.088645 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.088655    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-plugins-config-volume\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:50.088939 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.088669    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgkp\" (UniqueName: \"kubernetes.io/projected/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1-kube-api-access-9xgkp\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:23:50.089258 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.089235    2582 generic.go:356] "Generic (PLEG): container finished" podID="3ed87f14-37c6-4178-ba0c-aafba2fbf63c" containerID="1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911" exitCode=0
Sep 11 04:23:50.089346 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.089286    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"
Sep 11 04:23:50.089346 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.089321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww" event={"ID":"3ed87f14-37c6-4178-ba0c-aafba2fbf63c","Type":"ContainerDied","Data":"1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911"}
Sep 11 04:23:50.089439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.089366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww" event={"ID":"3ed87f14-37c6-4178-ba0c-aafba2fbf63c","Type":"ContainerDied","Data":"b3b89ed48d3a0b697bf7890f2fcafd237d37e667732839344fd4f671277e64ea"}
Sep 11 04:23:50.089439 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.089388    2582 scope.go:117] "RemoveContainer" containerID="1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911"
Sep 11 04:23:50.091012 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.090986    2582 generic.go:356] "Generic (PLEG): container finished" podID="975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" containerID="d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042" exitCode=0
Sep 11 04:23:50.091112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.091030    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"
Sep 11 04:23:50.091112 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.091065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz" event={"ID":"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1","Type":"ContainerDied","Data":"d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042"}
Sep 11 04:23:50.094115 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.091578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz" event={"ID":"975c1895-7f6a-44bb-aaa3-03bb1a7c47f1","Type":"ContainerDied","Data":"5d3edfca3bf5ec3dce50ef19a3c160ffdea24bff659a9a938862ffb02f6d2adf"}
Sep 11 04:23:50.107069 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.107051    2582 scope.go:117] "RemoveContainer" containerID="1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911"
Sep 11 04:23:50.107344 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:23:50.107327    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911\": container with ID starting with 1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911 not found: ID does not exist" containerID="1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911"
Sep 11 04:23:50.107395 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.107351    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911"} err="failed to get container status \"1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911\": rpc error: code = NotFound desc = could not find container \"1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911\": container with ID starting with 1cce002db199e1040234a5e0353c97847bb8e836a5e0ea426b5dee538a50e911 not found: ID does not exist"
Sep 11 04:23:50.107395 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.107367    2582 scope.go:117] "RemoveContainer" containerID="d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042"
Sep 11 04:23:50.115547 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.115527    2582 scope.go:117] "RemoveContainer" containerID="d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042"
Sep 11 04:23:50.115851 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:23:50.115831    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042\": container with ID starting with d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042 not found: ID does not exist" containerID="d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042"
Sep 11 04:23:50.115906 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.115859    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042"} err="failed to get container status \"d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042\": rpc error: code = NotFound desc = could not find container \"d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042\": container with ID starting with d3414eccd6f5a4b927a75c2ebb66313d338565bf194e7017b4a74e1396af0042 not found: ID does not exist"
Sep 11 04:23:50.119225 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.119203    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"]
Sep 11 04:23:50.124616 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.124592    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9342ed-69ff7d58f5-29zpz"]
Sep 11 04:23:50.136901 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.136874    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"]
Sep 11 04:23:50.141647 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.141590    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9342ed-55d6769f7d-7t6ww"]
Sep 11 04:23:50.275158 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.275122    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e" path="/var/lib/kubelet/pods/0f016f52-0e52-4cb1-89e3-f5ab6cf56c3e/volumes"
Sep 11 04:23:50.275482 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.275464    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed87f14-37c6-4178-ba0c-aafba2fbf63c" path="/var/lib/kubelet/pods/3ed87f14-37c6-4178-ba0c-aafba2fbf63c/volumes"
Sep 11 04:23:50.275833 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:23:50.275821    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" path="/var/lib/kubelet/pods/975c1895-7f6a-44bb-aaa3-03bb1a7c47f1/volumes"
Sep 11 04:24:46.685320 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:24:46.685284    2582 scope.go:117] "RemoveContainer" containerID="db14fad379a631d76c694d2b604a961279397cbd82dd187a082953b7aafb8fe1"
Sep 11 04:24:46.694389 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:24:46.694366    2582 scope.go:117] "RemoveContainer" containerID="86e475dfab78922437c903cbde2adca76ae0c404f3004601731afb5cc56dccd6"
Sep 11 04:25:00.626153 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626114    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645ff58b69-kfffs"]
Sep 11 04:25:00.626797 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626774    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed87f14-37c6-4178-ba0c-aafba2fbf63c" containerName="payload-pre-processing"
Sep 11 04:25:00.626880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626801    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed87f14-37c6-4178-ba0c-aafba2fbf63c" containerName="payload-pre-processing"
Sep 11 04:25:00.626880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626830    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" containerName="payload-processing"
Sep 11 04:25:00.626880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626837    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" containerName="payload-processing"
Sep 11 04:25:00.626880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626844    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b" containerName="revoke-keys"
Sep 11 04:25:00.626880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626851    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b" containerName="revoke-keys"
Sep 11 04:25:00.626880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626865    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4405beba-8ea0-47c8-a63e-2157673771b0" containerName="payload-processing"
Sep 11 04:25:00.626880 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626870    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4405beba-8ea0-47c8-a63e-2157673771b0" containerName="payload-processing"
Sep 11 04:25:00.627120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626964    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed87f14-37c6-4178-ba0c-aafba2fbf63c" containerName="payload-pre-processing"
Sep 11 04:25:00.627120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626977    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="975c1895-7f6a-44bb-aaa3-03bb1a7c47f1" containerName="payload-processing"
Sep 11 04:25:00.627120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626987    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb4ce6a-0ff2-4df0-81ba-d76ea8ee0d3b" containerName="revoke-keys"
Sep 11 04:25:00.627120 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.626993    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4405beba-8ea0-47c8-a63e-2157673771b0" containerName="payload-processing"
Sep 11 04:25:00.630092 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.630075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.633651 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.633629    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qwhqf\""
Sep 11 04:25:00.634809 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.634782    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 04:25:00.634933 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.634845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 04:25:00.634933 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.634909    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 04:25:00.647895 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.647865    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645ff58b69-kfffs"]
Sep 11 04:25:00.682341 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.682307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-metrics-certs\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.682529 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.682371    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-webhook-cert\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.682529 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.682455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqtxs\" (UniqueName: \"kubernetes.io/projected/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-kube-api-access-fqtxs\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.783129 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.783091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-webhook-cert\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.783307 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.783139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqtxs\" (UniqueName: \"kubernetes.io/projected/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-kube-api-access-fqtxs\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.783307 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.783227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-metrics-certs\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.785806 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.785776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-webhook-cert\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.785942 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.785785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-metrics-certs\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.791500 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.791467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqtxs\" (UniqueName: \"kubernetes.io/projected/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-kube-api-access-fqtxs\") pod \"maas-controller-645ff58b69-kfffs\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") " pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:00.940790 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:00.940753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:01.213241 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:01.213213    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645ff58b69-kfffs"]
Sep 11 04:25:01.215608 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:25:01.215582    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15424724_68b5_40dd_b5ef_1c62cf9ebd2b.slice/crio-8e2bd99a090b260ee8682c7d5847600135dfdbb37ad4836042e57d4b1be548d1 WatchSource:0}: Error finding container 8e2bd99a090b260ee8682c7d5847600135dfdbb37ad4836042e57d4b1be548d1: Status 404 returned error can't find the container with id 8e2bd99a090b260ee8682c7d5847600135dfdbb37ad4836042e57d4b1be548d1
Sep 11 04:25:01.217097 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:01.217077    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:25:01.486660 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:01.486548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645ff58b69-kfffs" event={"ID":"15424724-68b5-40dd-b5ef-1c62cf9ebd2b","Type":"ContainerStarted","Data":"8e2bd99a090b260ee8682c7d5847600135dfdbb37ad4836042e57d4b1be548d1"}
Sep 11 04:25:02.493724 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:02.493671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645ff58b69-kfffs" event={"ID":"15424724-68b5-40dd-b5ef-1c62cf9ebd2b","Type":"ContainerStarted","Data":"3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4"}
Sep 11 04:25:02.494122 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:02.493864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:02.511931 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:02.511875    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-645ff58b69-kfffs" podStartSLOduration=1.956375461 podStartE2EDuration="2.511834797s" totalImagesPullingTime="555.459336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:25:02.509919808 +0000 UTC m=+1876.774398179" watchObservedRunningTime="2026-09-11 04:25:02.511834797 +0000 UTC m=+1876.776313169"
Sep 11 04:25:13.501231 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:13.501146    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:22.865589 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:22.865531    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645ff58b69-kfffs"]
Sep 11 04:25:22.866080 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:22.865814    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-645ff58b69-kfffs" podUID="15424724-68b5-40dd-b5ef-1c62cf9ebd2b" containerName="manager" containerID="cri-o://3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4" gracePeriod=30
Sep 11 04:25:23.112571 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.112532    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:23.207309 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.207275    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-metrics-certs\") pod \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") "
Sep 11 04:25:23.207309 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.207314    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-webhook-cert\") pod \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") "
Sep 11 04:25:23.207518 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.207428    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqtxs\" (UniqueName: \"kubernetes.io/projected/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-kube-api-access-fqtxs\") pod \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\" (UID: \"15424724-68b5-40dd-b5ef-1c62cf9ebd2b\") "
Sep 11 04:25:23.209458 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.209426    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15424724-68b5-40dd-b5ef-1c62cf9ebd2b" (UID: "15424724-68b5-40dd-b5ef-1c62cf9ebd2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:25:23.209607 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.209480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "15424724-68b5-40dd-b5ef-1c62cf9ebd2b" (UID: "15424724-68b5-40dd-b5ef-1c62cf9ebd2b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:25:23.209607 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.209497    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-kube-api-access-fqtxs" (OuterVolumeSpecName: "kube-api-access-fqtxs") pod "15424724-68b5-40dd-b5ef-1c62cf9ebd2b" (UID: "15424724-68b5-40dd-b5ef-1c62cf9ebd2b"). InnerVolumeSpecName "kube-api-access-fqtxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:25:23.308616 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.308588    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqtxs\" (UniqueName: \"kubernetes.io/projected/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-kube-api-access-fqtxs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:25:23.308715 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.308618    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-metrics-certs\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:25:23.308715 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.308629    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15424724-68b5-40dd-b5ef-1c62cf9ebd2b-webhook-cert\") on node \"ip-10-0-138-179.ec2.internal\" DevicePath \"\""
Sep 11 04:25:23.610388 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.610296    2582 generic.go:356] "Generic (PLEG): container finished" podID="15424724-68b5-40dd-b5ef-1c62cf9ebd2b" containerID="3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4" exitCode=0
Sep 11 04:25:23.610388 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.610337    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645ff58b69-kfffs"
Sep 11 04:25:23.610388 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.610348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645ff58b69-kfffs" event={"ID":"15424724-68b5-40dd-b5ef-1c62cf9ebd2b","Type":"ContainerDied","Data":"3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4"}
Sep 11 04:25:23.610686 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.610401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645ff58b69-kfffs" event={"ID":"15424724-68b5-40dd-b5ef-1c62cf9ebd2b","Type":"ContainerDied","Data":"8e2bd99a090b260ee8682c7d5847600135dfdbb37ad4836042e57d4b1be548d1"}
Sep 11 04:25:23.610686 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.610435    2582 scope.go:117] "RemoveContainer" containerID="3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4"
Sep 11 04:25:23.620632 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.620611    2582 scope.go:117] "RemoveContainer" containerID="3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4"
Sep 11 04:25:23.620924 ip-10-0-138-179 kubenswrapper[2582]: E0911 04:25:23.620902    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4\": container with ID starting with 3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4 not found: ID does not exist" containerID="3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4"
Sep 11 04:25:23.620983 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.620934    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4"} err="failed to get container status \"3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4\": rpc error: code = NotFound desc = could not find container \"3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4\": container with ID starting with 3aa853b8e4581fe98728473f1cbc7e26828426314a2d6fc27903d8afeff821e4 not found: ID does not exist"
Sep 11 04:25:23.632023 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.631994    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645ff58b69-kfffs"]
Sep 11 04:25:23.634051 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:23.634024    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-645ff58b69-kfffs"]
Sep 11 04:25:24.275419 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:24.275381    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15424724-68b5-40dd-b5ef-1c62cf9ebd2b" path="/var/lib/kubelet/pods/15424724-68b5-40dd-b5ef-1c62cf9ebd2b/volumes"
Sep 11 04:25:46.728141 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:46.728095    2582 scope.go:117] "RemoveContainer" containerID="303dbbb3d619c7c5f1d195d714055625c592a20863366305056f5f9569dc4dc7"
Sep 11 04:25:46.737095 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:46.737074    2582 scope.go:117] "RemoveContainer" containerID="3568234edaa3125288201b285edf1ed43b1f2d6c0b0be1b2ae3fa51b355f6e75"
Sep 11 04:25:46.745462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:25:46.745439    2582 scope.go:117] "RemoveContainer" containerID="43cc5c5a364fdf439b3e19706d019fb4e87640c2b7d61cbaef438a7f7ffdc24e"
Sep 11 04:27:46.784493 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:27:46.784458    2582 scope.go:117] "RemoveContainer" containerID="61bff928d32b58ee52ebe27748a64abeae14ce8ceb4ef83906547eded0ee9ae4"
Sep 11 04:28:46.399067 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:28:46.398952    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:28:46.405301 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:28:46.405277    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdgl4_f2ec1032-0ce5-40e5-b2de-9779b8d88a1a/ovn-acl-logging/0.log"
Sep 11 04:28:46.805490 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:28:46.805453    2582 scope.go:117] "RemoveContainer" containerID="451b640c8c7e5e22b4c5f1341ea3ee050d7bb83130679d76a34189ebe43433c4"
Sep 11 04:29:46.825836 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:29:46.825756    2582 scope.go:117] "RemoveContainer" containerID="6322611297b6a8472053b2e15aaff7435e34abaa71462eba532a3abd5d859520"
Sep 11 04:29:46.834389 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:29:46.834367    2582 scope.go:117] "RemoveContainer" containerID="60a3875f8c63685ceefc00d74b18c012b7cd2ba5073d8fff459f8a15ba39a437"
Sep 11 04:29:59.591735 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:29:59.591694    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-tptr7_fbbebbc1-7a59-4011-8293-b64c404a1bc4/manager/0.log"
Sep 11 04:29:59.597762 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:29:59.597736    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-tptr7_fbbebbc1-7a59-4011-8293-b64c404a1bc4/copy-manifests/0.log"
Sep 11 04:30:00.143419 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.143383    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs"]
Sep 11 04:30:00.143811 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.143798    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15424724-68b5-40dd-b5ef-1c62cf9ebd2b" containerName="manager"
Sep 11 04:30:00.143859 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.143812    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="15424724-68b5-40dd-b5ef-1c62cf9ebd2b" containerName="manager"
Sep 11 04:30:00.143894 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.143887    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="15424724-68b5-40dd-b5ef-1c62cf9ebd2b" containerName="manager"
Sep 11 04:30:00.146813 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.146795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs"
Sep 11 04:30:00.149598 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.149573    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\""
Sep 11 04:30:00.155411 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.155386    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs"]
Sep 11 04:30:00.157866 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.157845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs"
Sep 11 04:30:00.183104 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.183074    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818335-rhkz2_7baa9c23-f733-4088-afff-86c1247fb59b/cleanup/0.log"
Sep 11 04:30:00.434252 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:00.434224    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs"]
Sep 11 04:30:00.436634 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:30:00.436605    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f30854_7687_4731_895b_ac19bf002251.slice/crio-006000f1f36b6bf0a6045787505df41a7d49cd27a7f98ec72a199db461992c2c WatchSource:0}: Error finding container 006000f1f36b6bf0a6045787505df41a7d49cd27a7f98ec72a199db461992c2c: Status 404 returned error can't find the container with id 006000f1f36b6bf0a6045787505df41a7d49cd27a7f98ec72a199db461992c2c
Sep 11 04:30:01.065955 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:01.065855    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs" event={"ID":"f7f30854-7687-4731-895b-ac19bf002251","Type":"ContainerStarted","Data":"742e6b88912c38e6d754e2e1c60d76bc46b2a6fd77bf4de9d6b5432990756b7d"}
Sep 11 04:30:01.065955 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:01.065907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs" event={"ID":"f7f30854-7687-4731-895b-ac19bf002251","Type":"ContainerStarted","Data":"006000f1f36b6bf0a6045787505df41a7d49cd27a7f98ec72a199db461992c2c"}
Sep 11 04:30:01.085044 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:01.084986    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs" podStartSLOduration=1.084967277 podStartE2EDuration="1.084967277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:30:01.083275341 +0000 UTC m=+2175.347753715" watchObservedRunningTime="2026-09-11 04:30:01.084967277 +0000 UTC m=+2175.349445648"
Sep 11 04:30:01.206667 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:01.206628    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-hmz2j_ba001e73-cea6-4982-9e26-d42428ffd640/manager/0.log"
Sep 11 04:30:02.072453 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:02.072414    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7f30854-7687-4731-895b-ac19bf002251" containerID="742e6b88912c38e6d754e2e1c60d76bc46b2a6fd77bf4de9d6b5432990756b7d" exitCode=0
Sep 11 04:30:02.072888 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:02.072497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs" event={"ID":"f7f30854-7687-4731-895b-ac19bf002251","Type":"ContainerDied","Data":"742e6b88912c38e6d754e2e1c60d76bc46b2a6fd77bf4de9d6b5432990756b7d"}
Sep 11 04:30:02.417263 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:02.417181    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f78fb9f44-7c4gt_d3675281-98a6-412e-8ab3-aa70b23d52fe/kube-auth-proxy/0.log"
Sep 11 04:30:02.782505 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:02.782462    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-52h68_a79f2440-749d-46d8-bf44-05cd7e933848/payload-pre-processing/0.log"
Sep 11 04:30:03.218459 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:03.218433    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs"
Sep 11 04:30:03.500827 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:03.500797    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-ggzcj_f9bd2e3a-d5b1-483b-9291-c2ce5481ccca/main/0.log"
Sep 11 04:30:03.615315 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:03.615279    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-gcng7_72cd726d-f835-4d5d-be78-c604e40e3d5b/main/0.log"
Sep 11 04:30:03.846744 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:03.846659    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffqwg7_febe73cb-b75b-48ca-83c3-39eaabc5d62c/main/0.log"
Sep 11 04:30:04.080764 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:04.080734    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-gq9bb_7375d0f4-e498-4241-98af-ac2749c1b056/main/0.log"
Sep 11 04:30:04.084751 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:04.084728    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs"
Sep 11 04:30:04.084867 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:04.084751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818350-qxghs" event={"ID":"f7f30854-7687-4731-895b-ac19bf002251","Type":"ContainerDied","Data":"006000f1f36b6bf0a6045787505df41a7d49cd27a7f98ec72a199db461992c2c"}
Sep 11 04:30:04.084867 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:04.084816    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006000f1f36b6bf0a6045787505df41a7d49cd27a7f98ec72a199db461992c2c"
Sep 11 04:30:13.005513 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:13.005479    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h4xhm_e11211d0-1d68-4ab1-9664-a997faf51785/global-pull-secret-syncer/0.log"
Sep 11 04:30:13.153302 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:13.153270    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kjt6x_d6b89fb3-4c0c-4014-8137-4d3ab6f6b587/konnectivity-agent/0.log"
Sep 11 04:30:13.232637 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:13.232597    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-179.ec2.internal_c473fc8668ce4f2af300af6f4bed0f91/haproxy/0.log"
Sep 11 04:30:17.636701 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:17.636615    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-hmz2j_ba001e73-cea6-4982-9e26-d42428ffd640/manager/0.log"
Sep 11 04:30:19.342652 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.342571    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6130d136-12f4-46f8-86f4-56086bcbebbf/alertmanager/0.log"
Sep 11 04:30:19.366255 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.366227    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6130d136-12f4-46f8-86f4-56086bcbebbf/config-reloader/0.log"
Sep 11 04:30:19.387928 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.387904    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6130d136-12f4-46f8-86f4-56086bcbebbf/kube-rbac-proxy-web/0.log"
Sep 11 04:30:19.413993 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.413970    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6130d136-12f4-46f8-86f4-56086bcbebbf/kube-rbac-proxy/0.log"
Sep 11 04:30:19.437749 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.437721    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6130d136-12f4-46f8-86f4-56086bcbebbf/kube-rbac-proxy-metric/0.log"
Sep 11 04:30:19.462238 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.462214    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6130d136-12f4-46f8-86f4-56086bcbebbf/prom-label-proxy/0.log"
Sep 11 04:30:19.485360 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.485331    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6130d136-12f4-46f8-86f4-56086bcbebbf/init-config-reloader/0.log"
Sep 11 04:30:19.552538 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.552513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zbjgj_0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8/kube-state-metrics/0.log"
Sep 11 04:30:19.572075 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.572044    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zbjgj_0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8/kube-rbac-proxy-main/0.log"
Sep 11 04:30:19.594821 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.594760    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zbjgj_0f8b3001-c5a8-48fb-9eec-cb3ca48cd9f8/kube-rbac-proxy-self/0.log"
Sep 11 04:30:19.624567 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.624521    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6777697447-sl8dp_9a24aaa1-764e-43ee-a16c-21022348b7bf/metrics-server/0.log"
Sep 11 04:30:19.665264 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.665227    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-2pk48_967439c6-58d9-41ae-8ad3-74def7677aa7/monitoring-plugin/0.log"
Sep 11 04:30:19.692354 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.692323    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49wg9_66968c6b-0d96-4040-b342-1de06758e994/node-exporter/0.log"
Sep 11 04:30:19.716250 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.716225    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49wg9_66968c6b-0d96-4040-b342-1de06758e994/kube-rbac-proxy/0.log"
Sep 11 04:30:19.748965 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.748943    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-49wg9_66968c6b-0d96-4040-b342-1de06758e994/init-textfile/0.log"
Sep 11 04:30:19.950278 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.950248    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-bsg64_7164ec5f-d1cf-41e7-84f1-28405bdaaaaf/kube-rbac-proxy-main/0.log"
Sep 11 04:30:19.972371 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.972345    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-bsg64_7164ec5f-d1cf-41e7-84f1-28405bdaaaaf/kube-rbac-proxy-self/0.log"
Sep 11 04:30:19.994903 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:19.994879    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-bsg64_7164ec5f-d1cf-41e7-84f1-28405bdaaaaf/openshift-state-metrics/0.log"
Sep 11 04:30:20.060981 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.060957    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4455fa2-c2e2-4127-8da7-0ecb2abed9dd/prometheus/0.log"
Sep 11 04:30:20.100395 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.100370    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4455fa2-c2e2-4127-8da7-0ecb2abed9dd/config-reloader/0.log"
Sep 11 04:30:20.129313 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.129289    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4455fa2-c2e2-4127-8da7-0ecb2abed9dd/thanos-sidecar/0.log"
Sep 11 04:30:20.153707 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.153687    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4455fa2-c2e2-4127-8da7-0ecb2abed9dd/kube-rbac-proxy-web/0.log"
Sep 11 04:30:20.171062 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.171039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4455fa2-c2e2-4127-8da7-0ecb2abed9dd/kube-rbac-proxy/0.log"
Sep 11 04:30:20.194103 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.194075    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4455fa2-c2e2-4127-8da7-0ecb2abed9dd/kube-rbac-proxy-thanos/0.log"
Sep 11 04:30:20.222293 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.222236    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4455fa2-c2e2-4127-8da7-0ecb2abed9dd/init-config-reloader/0.log"
Sep 11 04:30:20.256780 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.256760    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gjkfk_6cfdb15c-8061-4de8-87e6-69f8ed849338/prometheus-operator/0.log"
Sep 11 04:30:20.278917 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.278893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gjkfk_6cfdb15c-8061-4de8-87e6-69f8ed849338/kube-rbac-proxy/0.log"
Sep 11 04:30:20.312718 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.312697    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-tqzcx_38c27ccf-4920-49bd-a92c-b0969909718e/prometheus-operator-admission-webhook/0.log"
Sep 11 04:30:20.347631 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.347600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d97fb68cf-9qq2m_bd48999d-22f0-487b-ab80-17f843633e3f/telemeter-client/0.log"
Sep 11 04:30:20.373605 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.373582    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d97fb68cf-9qq2m_bd48999d-22f0-487b-ab80-17f843633e3f/reload/0.log"
Sep 11 04:30:20.402154 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.402130    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d97fb68cf-9qq2m_bd48999d-22f0-487b-ab80-17f843633e3f/kube-rbac-proxy/0.log"
Sep 11 04:30:20.449393 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.449363    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5479598577-fz8wj_1cd56357-d39e-47bd-8ddd-d716bb3064da/thanos-query/0.log"
Sep 11 04:30:20.497259 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.497179    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5479598577-fz8wj_1cd56357-d39e-47bd-8ddd-d716bb3064da/kube-rbac-proxy-web/0.log"
Sep 11 04:30:20.531492 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.531465    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5479598577-fz8wj_1cd56357-d39e-47bd-8ddd-d716bb3064da/kube-rbac-proxy/0.log"
Sep 11 04:30:20.561742 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.561720    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5479598577-fz8wj_1cd56357-d39e-47bd-8ddd-d716bb3064da/prom-label-proxy/0.log"
Sep 11 04:30:20.584994 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.584967    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5479598577-fz8wj_1cd56357-d39e-47bd-8ddd-d716bb3064da/kube-rbac-proxy-rules/0.log"
Sep 11 04:30:20.611500 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:20.611472    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5479598577-fz8wj_1cd56357-d39e-47bd-8ddd-d716bb3064da/kube-rbac-proxy-metrics/0.log"
Sep 11 04:30:23.172843 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.172803    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"]
Sep 11 04:30:23.173333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.173313    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f30854-7687-4731-895b-ac19bf002251" containerName="cleanup"
Sep 11 04:30:23.173333 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.173333    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f30854-7687-4731-895b-ac19bf002251" containerName="cleanup"
Sep 11 04:30:23.173485 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.173457    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f30854-7687-4731-895b-ac19bf002251" containerName="cleanup"
Sep 11 04:30:23.176472 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.176454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.179063 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.179037    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dxk5v\"/\"kube-root-ca.crt\""
Sep 11 04:30:23.179972 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.179945    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dxk5v\"/\"openshift-service-ca.crt\""
Sep 11 04:30:23.180032 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.179949    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dxk5v\"/\"default-dockercfg-gxcvx\""
Sep 11 04:30:23.187538 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.187514    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"]
Sep 11 04:30:23.311979 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.311948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-lib-modules\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.312069 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.311999    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ltt\" (UniqueName: \"kubernetes.io/projected/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-kube-api-access-x7ltt\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.312118 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.312095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-sys\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.312157 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.312124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-podres\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.312196 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.312184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-proc\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.318519 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.318491    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5dbb545bc8-8sdrq_59e4ecae-b7e0-4734-80a2-3bd4795a9c79/console/0.log"
Sep 11 04:30:23.412706 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-sys\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412706 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-podres\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-proc\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-lib-modules\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-sys\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ltt\" (UniqueName: \"kubernetes.io/projected/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-kube-api-access-x7ltt\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-podres\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-proc\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.412960 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.412938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-lib-modules\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.421269 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.421239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ltt\" (UniqueName: \"kubernetes.io/projected/644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8-kube-api-access-x7ltt\") pod \"perf-node-gather-daemonset-998jh\" (UID: \"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.487295 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.487216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:23.756630 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:23.756501    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"]
Sep 11 04:30:23.759531 ip-10-0-138-179 kubenswrapper[2582]: W0911 04:30:23.759501    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod644a3458_eb9a_4a5b_bad3_c0b8a7f23ff8.slice/crio-e5f211ea31ae473c892c60c6a0d4627e2e810a2b8dd73e5bcc2d5bff9c8d9276 WatchSource:0}: Error finding container e5f211ea31ae473c892c60c6a0d4627e2e810a2b8dd73e5bcc2d5bff9c8d9276: Status 404 returned error can't find the container with id e5f211ea31ae473c892c60c6a0d4627e2e810a2b8dd73e5bcc2d5bff9c8d9276
Sep 11 04:30:24.196823 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:24.196788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh" event={"ID":"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8","Type":"ContainerStarted","Data":"3765e4a72bf91e99e41d9553bac6f1628e8de01194e8626e8d326f04af3a7e2a"}
Sep 11 04:30:24.197206 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:24.196849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh" event={"ID":"644a3458-eb9a-4a5b-bad3-c0b8a7f23ff8","Type":"ContainerStarted","Data":"e5f211ea31ae473c892c60c6a0d4627e2e810a2b8dd73e5bcc2d5bff9c8d9276"}
Sep 11 04:30:24.197206 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:24.196975    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:24.213922 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:24.213875    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh" podStartSLOduration=1.213860633 podStartE2EDuration="1.213860633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 04:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:30:24.213330604 +0000 UTC m=+2198.477808975" watchObservedRunningTime="2026-09-11 04:30:24.213860633 +0000 UTC m=+2198.478339003"
Sep 11 04:30:25.279347 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:25.279319    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n5dbg_1ae43bed-2080-4a10-bb67-3c4fad5b5113/dns/0.log"
Sep 11 04:30:25.304182 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:25.304156    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n5dbg_1ae43bed-2080-4a10-bb67-3c4fad5b5113/kube-rbac-proxy/0.log"
Sep 11 04:30:25.402824 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:25.402798    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n9zq_070b965d-988d-4e5a-82bc-8fdac5b9ff71/dns-node-resolver/0.log"
Sep 11 04:30:26.056991 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:26.056951    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mljfk_8f6d2eaa-f594-4c22-921d-31d6f641b9f0/node-ca/0.log"
Sep 11 04:30:27.248605 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:27.248572    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f78fb9f44-7c4gt_d3675281-98a6-412e-8ab3-aa70b23d52fe/kube-auth-proxy/0.log"
Sep 11 04:30:27.355820 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:27.355789    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-52h68_a79f2440-749d-46d8-bf44-05cd7e933848/payload-pre-processing/0.log"
Sep 11 04:30:28.293989 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:28.293957    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f658l_fba0f58a-3507-49cb-9909-583301b4abfb/serve-healthcheck-canary/0.log"
Sep 11 04:30:28.858054 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:28.858025    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-klrjk_b13312eb-ce25-4e08-96be-6f1c1c5371f3/kube-rbac-proxy/0.log"
Sep 11 04:30:28.881462 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:28.881437    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-klrjk_b13312eb-ce25-4e08-96be-6f1c1c5371f3/exporter/0.log"
Sep 11 04:30:28.904372 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:28.904343    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-klrjk_b13312eb-ce25-4e08-96be-6f1c1c5371f3/extractor/0.log"
Sep 11 04:30:30.212889 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:30.212860    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-998jh"
Sep 11 04:30:32.028498 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:32.028463    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-tptr7_fbbebbc1-7a59-4011-8293-b64c404a1bc4/manager/0.log"
Sep 11 04:30:32.081190 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:32.081168    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-tptr7_fbbebbc1-7a59-4011-8293-b64c404a1bc4/copy-manifests/0.log"
Sep 11 04:30:33.521018 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:33.520984    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f8f45d65c-rxm5h_4dd94aeb-2aeb-4ee5-95a4-90601016d763/manager/0.log"
Sep 11 04:30:33.548319 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:33.548280    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5kn6d_baa1e1ba-a4ad-450a-b32a-8067cd923703/openshift-lws-operator/0.log"
Sep 11 04:30:33.836684 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:33.836651    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f78fb9f44-7c4gt_d3675281-98a6-412e-8ab3-aa70b23d52fe/kube-auth-proxy/0.log"
Sep 11 04:30:33.918969 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:33.918937    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-52h68_a79f2440-749d-46d8-bf44-05cd7e933848/payload-pre-processing/0.log"
Sep 11 04:30:35.680620 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:35.680590    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f78fb9f44-7c4gt_d3675281-98a6-412e-8ab3-aa70b23d52fe/kube-auth-proxy/0.log"
Sep 11 04:30:35.766811 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:35.766783    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-52h68_a79f2440-749d-46d8-bf44-05cd7e933848/payload-pre-processing/0.log"
Sep 11 04:30:39.975334 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:39.975304    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w55b_0c95d4e8-0310-4daf-a733-f10700f3bf22/kube-multus-additional-cni-plugins/0.log"
Sep 11 04:30:39.998136 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:39.998106    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w55b_0c95d4e8-0310-4daf-a733-f10700f3bf22/egress-router-binary-copy/0.log"
Sep 11 04:30:40.021083 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:40.021055    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w55b_0c95d4e8-0310-4daf-a733-f10700f3bf22/cni-plugins/0.log"
Sep 11 04:30:40.043059 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:40.043035    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w55b_0c95d4e8-0310-4daf-a733-f10700f3bf22/bond-cni-plugin/0.log"
Sep 11 04:30:40.064319 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:40.064292    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w55b_0c95d4e8-0310-4daf-a733-f10700f3bf22/routeoverride-cni/0.log"
Sep 11 04:30:40.086312 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:40.086292    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w55b_0c95d4e8-0310-4daf-a733-f10700f3bf22/whereabouts-cni-bincopy/0.log"
Sep 11 04:30:40.107965 ip-10-0-138-179 kubenswrapper[2582]: I0911 04:30:40.107937    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w55b_0c95d4e8-0310-4daf-a733-f10700f3bf22/whereabouts-cni/0.log"