Jul 07 15:13:02.477676 ip-10-0-131-161 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 15:13:02.856079 ip-10-0-131-161 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 15:13:02.856079 ip-10-0-131-161 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 15:13:02.856079 ip-10-0-131-161 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 15:13:02.856079 ip-10-0-131-161 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 15:13:02.856079 ip-10-0-131-161 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 15:13:02.856900 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.856717    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 15:13:02.863628 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863612    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:13:02.863628 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863628    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:13:02.863628 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863631    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863635    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863638    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863642    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863645    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863648    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863651    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863654    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863656    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863659    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863662    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863665    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863668    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863671    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863676    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863678    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863681    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863684    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863692    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863695    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:13:02.863726 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863698    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863700    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863703    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863706    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863709    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863712    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863714    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863717    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863719    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863722    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863724    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863727    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863730    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863733    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863736    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863739    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863743    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863745    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863748    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863750    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:13:02.864217 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863753    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863755    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863758    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863760    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863763    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863765    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863769    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863771    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863775    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863778    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863782    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863784    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863787    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863789    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863792    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863795    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863797    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863800    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863802    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863805    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:13:02.864696 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863807    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863810    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863812    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863815    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863818    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863820    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863822    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863825    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863828    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863832    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863834    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863837    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863840    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863842    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863847    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863850    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863853    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863856    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863858    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:13:02.865183 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863860    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863863    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863865    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863868    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863870    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863873    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863877    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863882    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863885    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863888    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863891    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863894    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863897    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863900    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863902    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863905    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863907    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.863910    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864696    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:13:02.865649 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864704    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864708    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864711    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864714    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864723    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864726    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864729    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864731    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864734    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864737    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864740    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864742    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864745    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864748    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864751    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864753    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864756    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864759    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864761    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864764    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:13:02.866124 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864767    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864770    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864773    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864775    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864778    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864782    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864786    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864789    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864792    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864795    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864798    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864800    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864803    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864806    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864809    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864811    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864815    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864818    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864821    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:13:02.866634 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864824    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864826    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864829    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864832    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864834    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864837    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864839    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864842    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864844    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864847    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864850    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864854    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864857    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864860    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864864    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864867    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864870    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864873    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864876    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864878    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:13:02.867119 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864881    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864883    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864886    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864889    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864891    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864894    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864897    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864900    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864902    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864905    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864908    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864911    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864913    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864916    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864919    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864923    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864925    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864928    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864931    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:13:02.867606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864933    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864936    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864939    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864941    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864944    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864947    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864950    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864953    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864955    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864958    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864961    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864963    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864966    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864968    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864971    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864974    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864977    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864979    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864982    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.864985    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:13:02.868057 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865375    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865380    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865383    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865412    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865417    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865419    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865422    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865425    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865427    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865430    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865433    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865435    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865438    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865440    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865446    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865450    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865454    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865457    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865460    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:13:02.868540 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865464    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865466    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865469    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865472    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865475    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865478    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865482    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865485    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865488    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865490    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865493    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865495    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865498    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865500    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865504    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865510    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865513    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865515    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865518    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865521    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:13:02.868984 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865523    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865526    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865528    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865531    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865534    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865536    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865539    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865543    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865545    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865548    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865551    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865553    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865556    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865558    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865561    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865563    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865566    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865568    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865571    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865573    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:13:02.869521 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865576    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865579    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865582    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865584    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865587    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865589    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865593    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865595    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865597    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865600    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865603    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865606    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865609    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865611    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865614    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865616    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865619    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865622    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865624    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865630    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:13:02.869987 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865633    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865636    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865639    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865642    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865645    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865648    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865651    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865654    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865657    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865659    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865662    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865666    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865668    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865671    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865674    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865676    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865679    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865682    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865684    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.865687    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:13:02.870475 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865757    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865764    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865772    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865779    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865785    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865790    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865796    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865802    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865805    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865808    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865811    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865815    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865818    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865824    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865827    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865831    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865834    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865837    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865841    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865844    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865847    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865850    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865854    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865857    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 15:13:02.870954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865860    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865863    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865867    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865870    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865872    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865875    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865878    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865895    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865900    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865903    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865906    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865909    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865913    2560 flags.go:64] FLAG: --enable-server="true"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865916    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865920    2560 flags.go:64] FLAG: --event-burst="100"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865923    2560 flags.go:64] FLAG: --event-qps="50"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865926    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865929    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865933    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865937    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865940    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865945    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865948    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865951    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865954    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 15:13:02.871532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865957    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865960    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865963    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865966    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865969    2560 flags.go:64] FLAG: --feature-gates=""
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865973    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865976    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865979    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865982    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865985    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865988    2560 flags.go:64] FLAG: --help="false"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865991    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-131-161.ec2.internal"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865995    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.865997    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866000    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866004    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866008    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866011    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866013    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866016    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866019    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866023    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866026    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866029    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 07 15:13:02.872182 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866031    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866034    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866038    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866041    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866045    2560 flags.go:64] FLAG: --lock-file=""
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866048    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866051    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866054    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866060    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866063    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866066    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866069    2560 flags.go:64] FLAG: --logging-format="text"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866072    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866075    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866078    2560 flags.go:64] FLAG: --manifest-url=""
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866081    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866085    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866088    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866092    2560 flags.go:64] FLAG: --max-pods="110"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866095    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866098    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866101    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866104    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866107    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866110    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 15:13:02.872809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866113    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866122    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866125    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866129    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866132    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866136    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866141    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866144    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866148    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866151    2560 flags.go:64] FLAG: --port="10250"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866154    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866159    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b311b71e7824ef89"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866162    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866165    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866168    2560 flags.go:64] FLAG: --register-node="true"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866171    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866174    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866177    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866180    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866183    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866187    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866190    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866193    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866196    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866199    2560 flags.go:64] FLAG: --runonce="false"
Jul 07 15:13:02.873432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866202    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866205    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866208    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866210    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866213    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866216    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866219    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866222    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866225    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866228    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866231    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866233    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866236    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866239    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866242    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866247    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866250    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866255    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866262    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866265    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866268    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866271    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866274    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866277    2560 flags.go:64] FLAG: --v="2"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866281    2560 flags.go:64] FLAG: --version="false"
Jul 07 15:13:02.874032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866286    2560 flags.go:64] FLAG: --vmodule=""
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866290    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.866293    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866426    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866431    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866434    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866437    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866440    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866442    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866445    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866448    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866451    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866454    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866456    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866459    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866462    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866465    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866467    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866470    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866472    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866475    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:13:02.874693 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866479    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866481    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866484    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866488    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866492    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866495    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866498    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866501    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866503    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866506    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866508    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866511    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866514    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866516    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866520    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866524    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866527    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866530    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866532    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:13:02.875191 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866535    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866538    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866541    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866544    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866546    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866549    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866551    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866554    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866556    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866559    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866563    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866566    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866569    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866572    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866575    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866578    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866582    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866587    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866591    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:13:02.875654 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866594    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866597    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866600    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866603    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866605    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866608    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866611    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866614    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866617    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866620    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866622    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866625    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866628    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866631    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866633    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866636    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866639    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866642    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866644    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:13:02.876154 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866647    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866650    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866653    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866655    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866658    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866661    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866663    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866666    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866668    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866671    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866676    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866680    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866682    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866685    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866688    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866691    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866693    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866696    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866698    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866701    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:13:02.876745 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866703    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866706    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866708    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.866711    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.867418    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.874190    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.874206    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.874984    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.874992    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875065    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875071    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875075    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875078    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875081    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875084    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:13:02.877214 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875087    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875090    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875093    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875096    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875099    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875102    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875106    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875110    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875113    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875115    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875118    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875120    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875123    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875126    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875128    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875131    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875134    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875136    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875139    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875141    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:13:02.877606 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875144    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875147    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875150    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875154    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875157    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875160    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875162    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875165    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875167    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875170    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875174    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875176    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875179    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875182    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875184    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875187    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875189    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875193    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875197    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:13:02.878075 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875200    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875203    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875206    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875209    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875211    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875214    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875217    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875220    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875222    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875225    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875228    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875230    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875232    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875235    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875237    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875241    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875244    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875247    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875250    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875253    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:13:02.878562 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875255    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875258    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875261    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875263    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875266    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875268    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875271    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875273    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875276    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875278    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875281    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875283    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875286    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875289    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875293    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875295    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875298    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875301    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875304    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875306    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:13:02.879049 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875309    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875311    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875314    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875317    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875319    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875322    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875324    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875328    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875330    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875333    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875336    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875338    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875341    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875343    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.875348    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 15:13:02.879615 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875490    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875496    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875499    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875502    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875505    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875507    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875511    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875514    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875516    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875519    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875522    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875526    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875529    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875532    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875535    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875538    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875541    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875545    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875548    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 15:13:02.880000 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875551    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875554    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875556    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875559    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875562    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875565    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875568    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875572    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875576    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875579    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875582    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875584    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875587    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875590    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875592    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875595    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875597    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875600    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875602    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875605    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 15:13:02.880475 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875608    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875610    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875612    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875615    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875617    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875620    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875622    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875625    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875627    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875629    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875632    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875634    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875637    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875639    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875642    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875644    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875647    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875649    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875652    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875654    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 15:13:02.880959 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875657    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875660    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875664    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875667    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875670    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875672    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875674    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875677    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875679    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875682    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875684    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875687    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875689    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875691    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875694    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875696    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875699    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875701    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875703    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875706    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 15:13:02.881444 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875709    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875712    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875714    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875716    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875719    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875721    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875723    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875726    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875728    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875732    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875734    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875737    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875740    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875742    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875745    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875750    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875753    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875755    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875758    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 15:13:02.881926 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:02.875760    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 15:13:02.882623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.875765    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 15:13:02.882623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.875882    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 15:13:02.882623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.878289    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 15:13:02.882623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.879756    2560 server.go:1019] "Starting client certificate rotation"
Jul 07 15:13:02.882623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.879862    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 15:13:02.882623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.879912    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 15:13:02.884866 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.884844    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 15:13:02.886040 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.886021    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 15:13:02.893763 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.893748    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 07 15:13:02.895894 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.895879    2560 log.go:25] "Validated CRI v1 image API"
Jul 07 15:13:02.897293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.897279    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 15:13:02.899586 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.899567    2560 fs.go:135] Filesystem UUIDs: map[116a1164-1f5f-4f06-b46c-d6ab6a8513bf:/dev/nvme0n1p4 5dc9201f-eeec-45b9-b130-7ae3a7c00f1d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 07 15:13:02.899635 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.899587    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 15:13:02.905109 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.904999    2560 manager.go:216] Machine: {Timestamp:2026-07-07 15:13:02.903772089 +0000 UTC m=+0.311880982 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099252 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279a88a28bda31d9eb20e7614ccab2 SystemUUID:ec279a88-a28b-da31-d9eb-20e7614ccab2 BootID:e992a779-4932-43b5-8140-1e786bdfbf72 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:18:40:f8:31:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:18:40:f8:31:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:76:94:cf:76:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 15:13:02.905109 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.905108    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 15:13:02.905217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.905192    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 15:13:02.908025 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.908003    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 15:13:02.908402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.908029    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 15:13:02.909539 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.909525    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 15:13:02.909578 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.909542    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 15:13:02.909578 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.909551    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 15:13:02.910274 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.910263    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 15:13:02.910896 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.910879    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 15:13:02.912529 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.912517    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 15:13:02.916286 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916273    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 15:13:02.916416 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916406    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 15:13:02.916458 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916438    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 15:13:02.916458 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916449    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 15:13:02.916562 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916477    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 15:13:02.916562 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916492    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 07 15:13:02.916562 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916509    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 15:13:02.917002 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.916990    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 15:13:02.917040 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.917015    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 15:13:02.919337 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.919320    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 15:13:02.921879 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.921861    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 15:13:02.921973 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.921902    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 15:13:02.923260 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923237    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 15:13:02.923352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923269    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 15:13:02.923352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923280    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 15:13:02.923352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923290    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 15:13:02.923352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923313    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 15:13:02.923352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923322    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 15:13:02.923352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923336    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 15:13:02.923352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923346    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 15:13:02.923623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923361    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 15:13:02.923623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923371    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 15:13:02.923623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923386    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 15:13:02.923623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.923397    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 15:13:02.924290 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.924276    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 15:13:02.924358 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.924296    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 15:13:02.927655 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:02.927633    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 15:13:02.927828 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.927810    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 15:13:02.927900 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:02.927833    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 15:13:02.928415 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.928400    2560 server.go:1268] "Started kubelet"
Jul 07 15:13:02.928509 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.928482    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 15:13:02.928606 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.928553    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 15:13:02.928641 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.928632    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 15:13:02.929547 ip-10-0-131-161 systemd[1]: Started Kubernetes Kubelet.
Jul 07 15:13:02.930681 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.930649    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 15:13:02.931450 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.931434    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 15:13:02.934585 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:02.933670    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-161.ec2.internal.18c00aab37ab2fdd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-161.ec2.internal,UID:ip-10-0-131-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-161.ec2.internal,},FirstTimestamp:2026-07-07 15:13:02.928367581 +0000 UTC m=+0.336476478,LastTimestamp:2026-07-07 15:13:02.928367581 +0000 UTC m=+0.336476478,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-161.ec2.internal,}"
Jul 07 15:13:02.934666 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:02.934607    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 15:13:02.943668 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.943645    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 15:13:02.943668 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.943659    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 15:13:02.943818 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:02.943765    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:02.944475 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.944457    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 15:13:02.944631 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.944611    2560 factory.go:55] Registering systemd factory
Jul 07 15:13:02.944704 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.944635    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 07 15:13:02.944704 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.944699    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 15:13:02.944821 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.944711    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 15:13:02.944901 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.944874    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 15:13:02.945086 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.945069    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 15:13:02.945086 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.944918    2560 factory.go:153] Registering CRI-O factory
Jul 07 15:13:02.945226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.945101    2560 factory.go:223] Registration of the crio container factory successfully
Jul 07 15:13:02.945226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.945148    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 15:13:02.945226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.945169    2560 factory.go:103] Registering Raw factory
Jul 07 15:13:02.945226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.945181    2560 manager.go:1196] Started watching for new ooms in manager
Jul 07 15:13:02.946001 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.945980    2560 manager.go:349] Starting recovery of all containers
Jul 07 15:13:02.946432 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:02.946389    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 15:13:02.946545 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:02.946485    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 15:13:02.958139 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.958114    2560 manager.go:354] Recovery completed
Jul 07 15:13:02.959684 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.959662    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwrsn"
Jul 07 15:13:02.962934 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.962920    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:13:02.964990 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.964973    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:13:02.965088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.965010    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:13:02.965088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.965025    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:13:02.965597 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.965581    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 15:13:02.965597 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.965596    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 15:13:02.965730 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.965596    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:13:02.965730 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.965617    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 15:13:02.966369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.966353    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:13:02.966449 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.966387    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:13:02.966449 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.966414    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:13:02.966697 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.966670    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwrsn"
Jul 07 15:13:02.966885 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.966865    2560 policy_none.go:49] "None policy: Start"
Jul 07 15:13:02.966885 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.966885    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 15:13:02.967026 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.966899    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 15:13:02.968188 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:02.968174    2560 policy_none.go:47] "Start"
Jul 07 15:13:03.004301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.004069    2560 manager.go:337] "Starting Device Plugin manager"
Jul 07 15:13:03.004417 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.004338    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 15:13:03.004417 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.004353    2560 server.go:87] "Starting device plugin registration server"
Jul 07 15:13:03.004676 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.004663    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 15:13:03.004710 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.004679    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 15:13:03.004839 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.004812    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 15:13:03.004940 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.004925    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 15:13:03.004940 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.004936    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 15:13:03.005598 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.005575    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 15:13:03.005701 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.005635    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.104947 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.104900    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:13:03.106135 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.106001    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:13:03.106135 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.106047    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:13:03.106135 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.106065    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:13:03.106135 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.106105    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.116381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.116359    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.116489 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.116385    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-161.ec2.internal\": node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.129052 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.129023    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 15:13:03.130435 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.130408    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 15:13:03.130584 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.130452    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 15:13:03.130584 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.130479    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 15:13:03.130584 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.130534    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 15:13:03.132889 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.132868    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:13:03.135663 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.135644    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.232084 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.232027    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal"]
Jul 07 15:13:03.232278 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.232130    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:13:03.236850 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.236829    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.237887 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.237869    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:13:03.237996 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.237902    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:13:03.237996 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.237913    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:13:03.239536 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.239522    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:13:03.239648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.239634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.239704 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.239667    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:13:03.240415 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.240397    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:13:03.240491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.240437    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:13:03.240491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.240460    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:13:03.240491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.240471    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:13:03.240491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.240443    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:13:03.240636 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.240518    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:13:03.241856 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.241842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.241906 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.241867    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 15:13:03.242764 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.242745    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 15:13:03.242885 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.242775    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 15:13:03.242885 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.242785    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeHasSufficientPID"
Jul 07 15:13:03.246091 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.246071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d35bdc223339f6f1674b941a36a6d17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal\" (UID: \"1d35bdc223339f6f1674b941a36a6d17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.246187 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.246098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6db363e1603ceda82be0f2b17302f46-config\") pod \"kube-apiserver-proxy-ip-10-0-131-161.ec2.internal\" (UID: \"d6db363e1603ceda82be0f2b17302f46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.246187 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.246116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d35bdc223339f6f1674b941a36a6d17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal\" (UID: \"1d35bdc223339f6f1674b941a36a6d17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.266496 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.266461    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-161.ec2.internal\" not found" node="ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.271306 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.271284    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-161.ec2.internal\" not found" node="ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.337957 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.337925    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.346718 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.346694    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d35bdc223339f6f1674b941a36a6d17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal\" (UID: \"1d35bdc223339f6f1674b941a36a6d17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.346791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.346741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d35bdc223339f6f1674b941a36a6d17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal\" (UID: \"1d35bdc223339f6f1674b941a36a6d17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.346831 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.346796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d35bdc223339f6f1674b941a36a6d17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal\" (UID: \"1d35bdc223339f6f1674b941a36a6d17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.346831 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.346818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6db363e1603ceda82be0f2b17302f46-config\") pod \"kube-apiserver-proxy-ip-10-0-131-161.ec2.internal\" (UID: \"d6db363e1603ceda82be0f2b17302f46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.346910 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.346844    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6db363e1603ceda82be0f2b17302f46-config\") pod \"kube-apiserver-proxy-ip-10-0-131-161.ec2.internal\" (UID: \"d6db363e1603ceda82be0f2b17302f46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.346910 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.346877    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d35bdc223339f6f1674b941a36a6d17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal\" (UID: \"1d35bdc223339f6f1674b941a36a6d17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.438633 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.438585    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.540328 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.540278    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.572158 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.572123    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.575602 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.575581    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:03.641232 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.641168    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.741663 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.741569    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.842031 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.841975    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.857902 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.857878    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:13:03.879287 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.879248    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 15:13:03.879477 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.879453    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 15:13:03.879531 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.879445    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 15:13:03.942537 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:03.942500    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-161.ec2.internal\" not found"
Jul 07 15:13:03.969043 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.968999    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 15:08:03 +0000 UTC" deadline="2027-12-31 23:26:20.065419586 +0000 UTC"
Jul 07 15:13:03.969043 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:03.969033    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13016h13m16.096389864s"
Jul 07 15:13:04.027737 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.027663    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:13:04.044041 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.044012    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:04.058811 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.058791    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 15:13:04.060627 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.060609    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal"
Jul 07 15:13:04.069763 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.069726    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 15:13:04.159651 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:04.159618    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6db363e1603ceda82be0f2b17302f46.slice/crio-688f69d323b697a820bdad163c7419b3430717754c1166628e7e8abdd240302c WatchSource:0}: Error finding container 688f69d323b697a820bdad163c7419b3430717754c1166628e7e8abdd240302c: Status 404 returned error can't find the container with id 688f69d323b697a820bdad163c7419b3430717754c1166628e7e8abdd240302c
Jul 07 15:13:04.166542 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.166526    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:13:04.202486 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:04.202454    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d35bdc223339f6f1674b941a36a6d17.slice/crio-8ced9cff9db64cb1e42436564b0995e9ec1d70d213c931174fd824de41b0a47c WatchSource:0}: Error finding container 8ced9cff9db64cb1e42436564b0995e9ec1d70d213c931174fd824de41b0a47c: Status 404 returned error can't find the container with id 8ced9cff9db64cb1e42436564b0995e9ec1d70d213c931174fd824de41b0a47c
Jul 07 15:13:04.452626 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.452600    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:13:04.683599 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.683566    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:13:04.918772 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.918729    2560 apiserver.go:52] "Watching apiserver"
Jul 07 15:13:04.926485 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.926457    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 15:13:04.927692 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.927659    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zggnx","openshift-multus/network-metrics-daemon-tb2xg","kube-system/konnectivity-agent-kslrc","openshift-dns/node-resolver-fjq24","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal","openshift-multus/multus-rsqlt","openshift-network-diagnostics/network-check-target-sfpgt","openshift-network-operator/iptables-alerter-bzkvh","openshift-ovn-kubernetes/ovnkube-node-sq2jk","kube-system/global-pull-secret-syncer-6njzf","kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6","openshift-cluster-node-tuning-operator/tuned-hqm52","openshift-image-registry/node-ca-cmfmg"]
Jul 07 15:13:04.929318 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.929299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:04.929442 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:04.929377    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:04.930492 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.930470    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:04.931580 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.931562    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:04.933321 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.933289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:04.933589 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.933563    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 15:13:04.933715 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.933695    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 15:13:04.934498 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.933786    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 15:13:04.934498 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.933787    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pgtx2\""
Jul 07 15:13:04.934498 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.934019    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 15:13:04.934498 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.934093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.934498 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.934102    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 15:13:04.934498 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.934215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hb9mw\""
Jul 07 15:13:04.935251 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.935223    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.935909 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.935887    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5dg7c\""
Jul 07 15:13:04.935999 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.935906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 15:13:04.936178 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.936160    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 15:13:04.936870 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.936808    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.938325 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.938173    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.939442 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.938815    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 15:13:04.939929 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.939905    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 15:13:04.940434 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.940411    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 15:13:04.940582 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.940565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 15:13:04.940820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.940800    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 15:13:04.940820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.940814    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xsh26\""
Jul 07 15:13:04.941073 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.941054    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7t64q\""
Jul 07 15:13:04.941136 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.941086    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 15:13:04.941833 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.941810    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 15:13:04.942088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.942070    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 15:13:04.942704 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.942684    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 15:13:04.942969 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.942952    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 15:13:04.943126 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.942687    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pmjnd\""
Jul 07 15:13:04.943247 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.943231    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 15:13:04.943607 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.943583    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 15:13:04.943958 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.943932    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 15:13:04.945368 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.944278    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n9n4h\""
Jul 07 15:13:04.945368 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.944317    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 15:13:04.945368 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.944629    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 15:13:04.946823 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.945636    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:04.946823 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:04.945730    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:04.946823 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.946510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:04.946823 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:04.946762    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:04.950855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.950833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:04.952108 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.952084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:04.953420 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.953369    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9j98h\""
Jul 07 15:13:04.953534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.953426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 15:13:04.953613 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.953587    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 15:13:04.953825 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.953809    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 15:13:04.954238 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954218    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 15:13:04.954465 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954447    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 15:13:04.954651 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cnibin\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.954729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfr4\" (UniqueName: \"kubernetes.io/projected/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-kube-api-access-5tfr4\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.954729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:04.954729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-k8s-cni-cncf-io\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.954890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.954890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-ovnkube-config\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.954890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-lib-modules\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.954890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-run-netns\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.954890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-log-socket\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.954890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-sys\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e95cd953-6426-481f-af3c-45c51721226f-konnectivity-ca\") pod \"konnectivity-agent-kslrc\" (UID: \"e95cd953-6426-481f-af3c-45c51721226f\") " pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-cni-multus\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-hostroot\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-daemon-config\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.954989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c354928-c869-4d16-afe4-d32dfdfed3a8-host-slash\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7tn\" (UniqueName: \"kubernetes.io/projected/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-kube-api-access-5c7tn\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-cni-bin\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-system-cni-dir\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysctl-d\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysctl-conf\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.955221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e95cd953-6426-481f-af3c-45c51721226f-agent-certs\") pod \"konnectivity-agent-kslrc\" (UID: \"e95cd953-6426-481f-af3c-45c51721226f\") " pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-cnibin\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-kubelet\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-var-lib-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-systemd\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-run\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-var-lib-kubelet\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-env-overrides\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-os-release\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-tmp\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-cni-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-ovn\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c354928-c869-4d16-afe4-d32dfdfed3a8-iptables-alerter-script\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-cni-binary-copy\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955623    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-systemd\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe6c135a-71bf-45ec-978a-697aeee6a751-ovn-node-metrics-cert\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:04.955883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-modprobe-d\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7k9\" (UniqueName: \"kubernetes.io/projected/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-kube-api-access-fr7k9\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955764    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-slash\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-node-log\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/220a5ee3-b52f-414a-8d29-7453eda2bfb9-hosts-file\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220a5ee3-b52f-414a-8d29-7453eda2bfb9-tmp-dir\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-system-cni-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/247e63ad-de91-47f1-bae3-c697a7e5f702-kubelet-config\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/220a5ee3-b52f-414a-8d29-7453eda2bfb9-kube-api-access-ks4zc\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.955997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysconfig\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-host\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnqh\" (UniqueName: \"kubernetes.io/projected/c5001893-317d-42d4-afc4-3392d1d442ea-kube-api-access-sbnqh\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956074    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-os-release\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.956728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-systemd-units\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-etc-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-cni-bin\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-multus-certs\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-etc-kubernetes\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-kubernetes\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-tuned\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhffn\" (UniqueName: \"kubernetes.io/projected/1c354928-c869-4d16-afe4-d32dfdfed3a8-kube-api-access-xhffn\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-socket-dir-parent\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-netns\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-kubelet\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-conf-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-cni-netd\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-ovnkube-script-lib\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwll8\" (UniqueName: \"kubernetes.io/projected/fe6c135a-71bf-45ec-978a-697aeee6a751-kube-api-access-cwll8\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:04.957655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.956630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/247e63ad-de91-47f1-bae3-c697a7e5f702-dbus\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:04.958659 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.957201    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 15:13:04.958659 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.957268    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 15:13:04.958659 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.957268    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zdc2v\""
Jul 07 15:13:04.984979 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.984857    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6pxf7"
Jul 07 15:13:04.995058 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:04.994588    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6pxf7"
Jul 07 15:13:05.045927 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.045891    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-sys-fs\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c354928-c869-4d16-afe4-d32dfdfed3a8-host-slash\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7tn\" (UniqueName: \"kubernetes.io/projected/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-kube-api-access-5c7tn\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-cni-bin\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-system-cni-dir\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057756    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysctl-d\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057776    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057776    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c354928-c869-4d16-afe4-d32dfdfed3a8-host-slash\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysctl-conf\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057800    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-cni-bin\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e95cd953-6426-481f-af3c-45c51721226f-agent-certs\") pod \"konnectivity-agent-kslrc\" (UID: \"e95cd953-6426-481f-af3c-45c51721226f\") " pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057822    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-system-cni-dir\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-cnibin\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057902    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-cnibin\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.058117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-kubelet\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-var-lib-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057970    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysctl-conf\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057976    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-kubelet\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057936    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysctl-d\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.057988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-systemd\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-run\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058017    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-var-lib-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058054    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-var-lib-kubelet\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058114    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-var-lib-kubelet\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058094    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-systemd\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-env-overrides\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-os-release\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058164    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-run\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-tmp\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvwg\" (UniqueName: \"kubernetes.io/projected/5d0644a9-b118-45fa-9e14-77ee2cd8c242-kube-api-access-hjvwg\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-cni-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058233    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 15:13:05.059104 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-ovn\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c354928-c869-4d16-afe4-d32dfdfed3a8-iptables-alerter-script\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058299    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-os-release\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-cni-binary-copy\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-systemd\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058361    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-cni-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe6c135a-71bf-45ec-978a-697aeee6a751-ovn-node-metrics-cert\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-modprobe-d\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7k9\" (UniqueName: \"kubernetes.io/projected/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-kube-api-access-fr7k9\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-etc-selinux\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-slash\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058547    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058570    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-ovn\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-node-log\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058658    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/220a5ee3-b52f-414a-8d29-7453eda2bfb9-hosts-file\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:05.060078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220a5ee3-b52f-414a-8d29-7453eda2bfb9-tmp-dir\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-system-cni-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058712    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-env-overrides\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/247e63ad-de91-47f1-bae3-c697a7e5f702-kubelet-config\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/220a5ee3-b52f-414a-8d29-7453eda2bfb9-kube-api-access-ks4zc\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058769    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysconfig\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-host\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnqh\" (UniqueName: \"kubernetes.io/projected/c5001893-317d-42d4-afc4-3392d1d442ea-kube-api-access-sbnqh\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-os-release\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.058885    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.058947    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret podName:247e63ad-de91-47f1-bae3-c697a7e5f702 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:05.558927963 +0000 UTC m=+2.967036848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret") pod "global-pull-secret-syncer-6njzf" (UID: "247e63ad-de91-47f1-bae3-c697a7e5f702") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d0644a9-b118-45fa-9e14-77ee2cd8c242-host\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.060921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-systemd-units\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059070    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-etc-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059126    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/220a5ee3-b52f-414a-8d29-7453eda2bfb9-hosts-file\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-device-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-cni-bin\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-multus-certs\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-etc-kubernetes\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-kubernetes\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-tuned\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhffn\" (UniqueName: \"kubernetes.io/projected/1c354928-c869-4d16-afe4-d32dfdfed3a8-kube-api-access-xhffn\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-socket-dir-parent\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-netns\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-kubelet\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-conf-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059401    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220a5ee3-b52f-414a-8d29-7453eda2bfb9-tmp-dir\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:05.061820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-cni-netd\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-ovnkube-script-lib\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058816    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-node-log\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.058864    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-slash\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059689    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-system-cni-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059706    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/247e63ad-de91-47f1-bae3-c697a7e5f702-kubelet-config\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059733    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-systemd-units\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059769    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-etc-openvswitch\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.059841    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.059897    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs podName:c5001893-317d-42d4-afc4-3392d1d442ea nodeName:}" failed. No retries permitted until 2026-07-07 15:13:05.559880039 +0000 UTC m=+2.967988922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs") pod "network-metrics-daemon-tb2xg" (UID: "c5001893-317d-42d4-afc4-3392d1d442ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059917    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c354928-c869-4d16-afe4-d32dfdfed3a8-iptables-alerter-script\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwll8\" (UniqueName: \"kubernetes.io/projected/fe6c135a-71bf-45ec-978a-697aeee6a751-kube-api-access-cwll8\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059965    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-multus-certs\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/247e63ad-de91-47f1-bae3-c697a7e5f702-dbus\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059999    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-netns\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cnibin\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfr4\" (UniqueName: \"kubernetes.io/projected/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-kube-api-access-5tfr4\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.062588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060041    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-kubernetes\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-k8s-cni-cncf-io\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-socket-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060140    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-sysconfig\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-registration-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060188    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-host\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-ovnkube-config\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-lib-modules\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.059935    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-cni-bin\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87wr\" (UniqueName: \"kubernetes.io/projected/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-kube-api-access-z87wr\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-run-netns\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-log-socket\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-sys\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e95cd953-6426-481f-af3c-45c51721226f-konnectivity-ca\") pod \"konnectivity-agent-kslrc\" (UID: \"e95cd953-6426-481f-af3c-45c51721226f\") " pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-cni-multus\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.063470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-hostroot\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-daemon-config\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060594    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d0644a9-b118-45fa-9e14-77ee2cd8c242-serviceca\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060682    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-ovnkube-script-lib\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060733    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-run-systemd\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060752    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-lib-modules\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060806    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-os-release\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060860    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-etc-kubernetes\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.060975    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/247e63ad-de91-47f1-bae3-c697a7e5f702-dbus\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:05.064314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.061018    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cnibin\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.065317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.065285    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.065735 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.065708    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-modprobe-d\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.065824 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.065737    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-cni-netd\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.065878 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.065851    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-cni-binary-copy\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.066282 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.066251    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe6c135a-71bf-45ec-978a-697aeee6a751-ovn-node-metrics-cert\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.066378    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e95cd953-6426-481f-af3c-45c51721226f-agent-certs\") pod \"konnectivity-agent-kslrc\" (UID: \"e95cd953-6426-481f-af3c-45c51721226f\") " pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.066540    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-etc-tuned\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.066881    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067195    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067403    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-sys\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067529    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-run-k8s-cni-cncf-io\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067664    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e95cd953-6426-481f-af3c-45c51721226f-konnectivity-ca\") pod \"konnectivity-agent-kslrc\" (UID: \"e95cd953-6426-481f-af3c-45c51721226f\") " pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067666    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-hostroot\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067693    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-daemon-config\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067699    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-log-socket\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067735    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6c135a-71bf-45ec-978a-697aeee6a751-host-run-netns\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067747    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-conf-dir\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067789    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-kubelet\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067832    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-host-var-lib-cni-multus\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.067858    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-multus-socket-dir-parent\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.068323    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-tmp\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.068327    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe6c135a-71bf-45ec-978a-697aeee6a751-ovnkube-config\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.071046 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.070024    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7tn\" (UniqueName: \"kubernetes.io/projected/8048ebf5-390f-44f1-9ba9-a3a7c46ad771-kube-api-access-5c7tn\") pod \"multus-rsqlt\" (UID: \"8048ebf5-390f-44f1-9ba9-a3a7c46ad771\") " pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.075645 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.075620    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7k9\" (UniqueName: \"kubernetes.io/projected/4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c-kube-api-access-fr7k9\") pod \"tuned-hqm52\" (UID: \"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c\") " pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.078662 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.078630    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnqh\" (UniqueName: \"kubernetes.io/projected/c5001893-317d-42d4-afc4-3392d1d442ea-kube-api-access-sbnqh\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:05.080475 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.080450    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwll8\" (UniqueName: \"kubernetes.io/projected/fe6c135a-71bf-45ec-978a-697aeee6a751-kube-api-access-cwll8\") pod \"ovnkube-node-sq2jk\" (UID: \"fe6c135a-71bf-45ec-978a-697aeee6a751\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.080564 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.080474    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:13:05.080564 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.080496    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:13:05.080564 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.080512    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mwk2v for pod openshift-network-diagnostics/network-check-target-sfpgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:05.080733 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.080587    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v podName:58e30377-aff5-40b7-918a-3f0ee9c6ef74 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:05.580566966 +0000 UTC m=+2.988675870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mwk2v" (UniqueName: "kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v") pod "network-check-target-sfpgt" (UID: "58e30377-aff5-40b7-918a-3f0ee9c6ef74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:05.082168 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.082145    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4zc\" (UniqueName: \"kubernetes.io/projected/220a5ee3-b52f-414a-8d29-7453eda2bfb9-kube-api-access-ks4zc\") pod \"node-resolver-fjq24\" (UID: \"220a5ee3-b52f-414a-8d29-7453eda2bfb9\") " pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:05.083533 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.083508    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfr4\" (UniqueName: \"kubernetes.io/projected/9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd-kube-api-access-5tfr4\") pod \"multus-additional-cni-plugins-zggnx\" (UID: \"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd\") " pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.089866 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.089820    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhffn\" (UniqueName: \"kubernetes.io/projected/1c354928-c869-4d16-afe4-d32dfdfed3a8-kube-api-access-xhffn\") pod \"iptables-alerter-bzkvh\" (UID: \"1c354928-c869-4d16-afe4-d32dfdfed3a8\") " pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:05.138229 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.138140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal" event={"ID":"1d35bdc223339f6f1674b941a36a6d17","Type":"ContainerStarted","Data":"8ced9cff9db64cb1e42436564b0995e9ec1d70d213c931174fd824de41b0a47c"}
Jul 07 15:13:05.140338 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.140256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal" event={"ID":"d6db363e1603ceda82be0f2b17302f46","Type":"ContainerStarted","Data":"688f69d323b697a820bdad163c7419b3430717754c1166628e7e8abdd240302c"}
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d0644a9-b118-45fa-9e14-77ee2cd8c242-host\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-device-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-socket-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-registration-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87wr\" (UniqueName: \"kubernetes.io/projected/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-kube-api-access-z87wr\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d0644a9-b118-45fa-9e14-77ee2cd8c242-serviceca\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-sys-fs\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvwg\" (UniqueName: \"kubernetes.io/projected/5d0644a9-b118-45fa-9e14-77ee2cd8c242-kube-api-access-hjvwg\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-etc-selinux\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161805    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-registration-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161865    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d0644a9-b118-45fa-9e14-77ee2cd8c242-host\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161887    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-etc-selinux\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161913    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.161960    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-device-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.162084    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-socket-dir\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.162314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.162140    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-sys-fs\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.163226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.162582    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d0644a9-b118-45fa-9e14-77ee2cd8c242-serviceca\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.172448 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.172310    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvwg\" (UniqueName: \"kubernetes.io/projected/5d0644a9-b118-45fa-9e14-77ee2cd8c242-kube-api-access-hjvwg\") pod \"node-ca-cmfmg\" (UID: \"5d0644a9-b118-45fa-9e14-77ee2cd8c242\") " pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.172693 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.172654    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87wr\" (UniqueName: \"kubernetes.io/projected/91a18b81-7085-4b8c-8ecf-0b3cf6d413af-kube-api-access-z87wr\") pod \"aws-ebs-csi-driver-node-cddt6\" (UID: \"91a18b81-7085-4b8c-8ecf-0b3cf6d413af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.252171 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.252134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzkvh"
Jul 07 15:13:05.260434 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.260384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:05.272460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.272428    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fjq24"
Jul 07 15:13:05.279667 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.279647    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zggnx"
Jul 07 15:13:05.289256 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.289236    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsqlt"
Jul 07 15:13:05.299007 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.298982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:05.306858 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.306838    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqm52"
Jul 07 15:13:05.316829 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.316801    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmfmg"
Jul 07 15:13:05.325523 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.325494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6"
Jul 07 15:13:05.564742 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.564642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:05.564742 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.564697    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:05.564971 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.564807    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:05.564971 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.564807    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:05.564971 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.564871    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs podName:c5001893-317d-42d4-afc4-3392d1d442ea nodeName:}" failed. No retries permitted until 2026-07-07 15:13:06.564851821 +0000 UTC m=+3.972960703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs") pod "network-metrics-daemon-tb2xg" (UID: "c5001893-317d-42d4-afc4-3392d1d442ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:05.564971 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.564902    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret podName:247e63ad-de91-47f1-bae3-c697a7e5f702 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:06.564890558 +0000 UTC m=+3.972999442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret") pod "global-pull-secret-syncer-6njzf" (UID: "247e63ad-de91-47f1-bae3-c697a7e5f702") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:05.665682 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.665639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:05.665873 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.665807    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:13:05.665873 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.665827    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:13:05.665873 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.665839    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mwk2v for pod openshift-network-diagnostics/network-check-target-sfpgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:05.666016 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:05.665905    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v podName:58e30377-aff5-40b7-918a-3f0ee9c6ef74 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:06.665887694 +0000 UTC m=+4.073996590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwk2v" (UniqueName: "kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v") pod "network-check-target-sfpgt" (UID: "58e30377-aff5-40b7-918a-3f0ee9c6ef74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:05.997080 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.997036    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:08:05 +0000 UTC" deadline="2027-12-25 12:27:45.890702012 +0000 UTC"
Jul 07 15:13:05.997080 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:05.997075    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12861h14m39.893631473s"
Jul 07 15:13:06.130911 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:06.130879    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:06.131077 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.130994    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:06.273846 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.273822    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220a5ee3_b52f_414a_8d29_7453eda2bfb9.slice/crio-ff4cb4a0ceb71852a16455dd47a5dea45f0d0f389b17027e5ffb4960a6972a75 WatchSource:0}: Error finding container ff4cb4a0ceb71852a16455dd47a5dea45f0d0f389b17027e5ffb4960a6972a75: Status 404 returned error can't find the container with id ff4cb4a0ceb71852a16455dd47a5dea45f0d0f389b17027e5ffb4960a6972a75
Jul 07 15:13:06.275339 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.275309    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6c135a_71bf_45ec_978a_697aeee6a751.slice/crio-e6a81eb1d7a8ec25ffb51cf555d3b10a30b1c9743e01a8517dc2a4d599170fc8 WatchSource:0}: Error finding container e6a81eb1d7a8ec25ffb51cf555d3b10a30b1c9743e01a8517dc2a4d599170fc8: Status 404 returned error can't find the container with id e6a81eb1d7a8ec25ffb51cf555d3b10a30b1c9743e01a8517dc2a4d599170fc8
Jul 07 15:13:06.278921 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.278895    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0644a9_b118_45fa_9e14_77ee2cd8c242.slice/crio-dfc64bd83c8df0606aa2145e1cd6c08f05255c278d0b06d9db60db9b4c9c48fd WatchSource:0}: Error finding container dfc64bd83c8df0606aa2145e1cd6c08f05255c278d0b06d9db60db9b4c9c48fd: Status 404 returned error can't find the container with id dfc64bd83c8df0606aa2145e1cd6c08f05255c278d0b06d9db60db9b4c9c48fd
Jul 07 15:13:06.280347 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.280271    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aeb6b5b_929c_4c7d_bcbb_2273b0102f8c.slice/crio-5b25932ee329626d3151915b74cdf74b1e48afca73537ecbb70530bc1da4633d WatchSource:0}: Error finding container 5b25932ee329626d3151915b74cdf74b1e48afca73537ecbb70530bc1da4633d: Status 404 returned error can't find the container with id 5b25932ee329626d3151915b74cdf74b1e48afca73537ecbb70530bc1da4633d
Jul 07 15:13:06.281497 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.281465    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a18b81_7085_4b8c_8ecf_0b3cf6d413af.slice/crio-d63a6d08b6a11b9e6760db6298cf282e5ae3383475d216970a3888768d80d05f WatchSource:0}: Error finding container d63a6d08b6a11b9e6760db6298cf282e5ae3383475d216970a3888768d80d05f: Status 404 returned error can't find the container with id d63a6d08b6a11b9e6760db6298cf282e5ae3383475d216970a3888768d80d05f
Jul 07 15:13:06.283210 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.282648    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8048ebf5_390f_44f1_9ba9_a3a7c46ad771.slice/crio-82560e0a97cb69e7b6ed2024155bfb0e260ddf94bbb55f638fb7a34745c860d9 WatchSource:0}: Error finding container 82560e0a97cb69e7b6ed2024155bfb0e260ddf94bbb55f638fb7a34745c860d9: Status 404 returned error can't find the container with id 82560e0a97cb69e7b6ed2024155bfb0e260ddf94bbb55f638fb7a34745c860d9
Jul 07 15:13:06.284288 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.283713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a47aeac_5a3d_4dc2_97d3_554bfbd1c8bd.slice/crio-357d0b5aaa467f7394b3d5e7c7c801500f0bc0ef40e15c7470b7bd416a0695af WatchSource:0}: Error finding container 357d0b5aaa467f7394b3d5e7c7c801500f0bc0ef40e15c7470b7bd416a0695af: Status 404 returned error can't find the container with id 357d0b5aaa467f7394b3d5e7c7c801500f0bc0ef40e15c7470b7bd416a0695af
Jul 07 15:13:06.285366 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:06.285342    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c354928_c869_4d16_afe4_d32dfdfed3a8.slice/crio-9327579984c68c037c8511df218f030811b68fe062c09ddaef4a046bde6130bf WatchSource:0}: Error finding container 9327579984c68c037c8511df218f030811b68fe062c09ddaef4a046bde6130bf: Status 404 returned error can't find the container with id 9327579984c68c037c8511df218f030811b68fe062c09ddaef4a046bde6130bf
Jul 07 15:13:06.572242 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:06.572072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:06.572242 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.572232    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:06.572378 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:06.572280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:06.572378 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.572302    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs podName:c5001893-317d-42d4-afc4-3392d1d442ea nodeName:}" failed. No retries permitted until 2026-07-07 15:13:08.572284723 +0000 UTC m=+5.980393603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs") pod "network-metrics-daemon-tb2xg" (UID: "c5001893-317d-42d4-afc4-3392d1d442ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:06.572378 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.572340    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:06.572378 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.572378    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret podName:247e63ad-de91-47f1-bae3-c697a7e5f702 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:08.572367448 +0000 UTC m=+5.980476333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret") pod "global-pull-secret-syncer-6njzf" (UID: "247e63ad-de91-47f1-bae3-c697a7e5f702") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:06.673861 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:06.673740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:06.674017 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.674002    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:13:06.674147 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.674028    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:13:06.674147 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.674042    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mwk2v for pod openshift-network-diagnostics/network-check-target-sfpgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:06.674348 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:06.674211    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v podName:58e30377-aff5-40b7-918a-3f0ee9c6ef74 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:08.674192422 +0000 UTC m=+6.082301317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwk2v" (UniqueName: "kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v") pod "network-check-target-sfpgt" (UID: "58e30377-aff5-40b7-918a-3f0ee9c6ef74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:06.724039 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:06.724006    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 15:13:06.998072 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:06.998012    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:08:05 +0000 UTC" deadline="2028-03-22 16:19:51.94551987 +0000 UTC"
Jul 07 15:13:06.998072 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:06.998072    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14977h6m44.947452144s"
Jul 07 15:13:07.131402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.131352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:07.131674 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:07.131496    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:07.131896 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.131875    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:07.132004 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:07.131968    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:07.157408 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.155756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmfmg" event={"ID":"5d0644a9-b118-45fa-9e14-77ee2cd8c242","Type":"ContainerStarted","Data":"dfc64bd83c8df0606aa2145e1cd6c08f05255c278d0b06d9db60db9b4c9c48fd"}
Jul 07 15:13:07.161716 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.161599    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"e6a81eb1d7a8ec25ffb51cf555d3b10a30b1c9743e01a8517dc2a4d599170fc8"}
Jul 07 15:13:07.166060 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.166028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal" event={"ID":"d6db363e1603ceda82be0f2b17302f46","Type":"ContainerStarted","Data":"9fbd967d97918d37d39185ff722cea49aefabedda575114bd2d8dd57908dbaab"}
Jul 07 15:13:07.169282 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.169254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kslrc" event={"ID":"e95cd953-6426-481f-af3c-45c51721226f","Type":"ContainerStarted","Data":"94148e78a72876c264009ab48ca917569dd6f31fd65329f70011debaadd61143"}
Jul 07 15:13:07.174011 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.173934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzkvh" event={"ID":"1c354928-c869-4d16-afe4-d32dfdfed3a8","Type":"ContainerStarted","Data":"9327579984c68c037c8511df218f030811b68fe062c09ddaef4a046bde6130bf"}
Jul 07 15:13:07.179493 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.179358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerStarted","Data":"357d0b5aaa467f7394b3d5e7c7c801500f0bc0ef40e15c7470b7bd416a0695af"}
Jul 07 15:13:07.183410 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.183343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsqlt" event={"ID":"8048ebf5-390f-44f1-9ba9-a3a7c46ad771","Type":"ContainerStarted","Data":"82560e0a97cb69e7b6ed2024155bfb0e260ddf94bbb55f638fb7a34745c860d9"}
Jul 07 15:13:07.200609 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.200581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjq24" event={"ID":"220a5ee3-b52f-414a-8d29-7453eda2bfb9","Type":"ContainerStarted","Data":"ff4cb4a0ceb71852a16455dd47a5dea45f0d0f389b17027e5ffb4960a6972a75"}
Jul 07 15:13:07.206116 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.205383    2560 generic.go:356] "Generic (PLEG): container finished" podID="1d35bdc223339f6f1674b941a36a6d17" containerID="5bedd9e6c20cbe2cd9204ca32f62f47c15132ce922970bbfdc5cfc15e987c985" exitCode=0
Jul 07 15:13:07.206216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.206183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal" event={"ID":"1d35bdc223339f6f1674b941a36a6d17","Type":"ContainerDied","Data":"5bedd9e6c20cbe2cd9204ca32f62f47c15132ce922970bbfdc5cfc15e987c985"}
Jul 07 15:13:07.210118 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.209308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqm52" event={"ID":"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c","Type":"ContainerStarted","Data":"5b25932ee329626d3151915b74cdf74b1e48afca73537ecbb70530bc1da4633d"}
Jul 07 15:13:07.218260 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.218237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6" event={"ID":"91a18b81-7085-4b8c-8ecf-0b3cf6d413af","Type":"ContainerStarted","Data":"d63a6d08b6a11b9e6760db6298cf282e5ae3383475d216970a3888768d80d05f"}
Jul 07 15:13:07.223866 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:07.223814    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-161.ec2.internal" podStartSLOduration=3.223796287 podStartE2EDuration="3.223796287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:07.183219828 +0000 UTC m=+4.591328734" watchObservedRunningTime="2026-07-07 15:13:07.223796287 +0000 UTC m=+4.631905188"
Jul 07 15:13:08.130926 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:08.130891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:08.131402 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.131027    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:08.236649 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:08.236614    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal" event={"ID":"1d35bdc223339f6f1674b941a36a6d17","Type":"ContainerStarted","Data":"adfbc9de53af5f8e418f9c18adf41c67ed4eae0f6a68ebdf246876534b8bbf01"}
Jul 07 15:13:08.591217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:08.590457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:08.591217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:08.590519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:08.591217 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.590680    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:08.591217 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.590762    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs podName:c5001893-317d-42d4-afc4-3392d1d442ea nodeName:}" failed. No retries permitted until 2026-07-07 15:13:12.590743439 +0000 UTC m=+9.998852334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs") pod "network-metrics-daemon-tb2xg" (UID: "c5001893-317d-42d4-afc4-3392d1d442ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:08.591217 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.591137    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:08.591217 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.591181    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret podName:247e63ad-de91-47f1-bae3-c697a7e5f702 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:12.59116726 +0000 UTC m=+9.999276145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret") pod "global-pull-secret-syncer-6njzf" (UID: "247e63ad-de91-47f1-bae3-c697a7e5f702") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:08.692469 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:08.691619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:08.692469 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.691935    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:13:08.692469 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.691955    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:13:08.692469 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.691969    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mwk2v for pod openshift-network-diagnostics/network-check-target-sfpgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:08.692469 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:08.692037    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v podName:58e30377-aff5-40b7-918a-3f0ee9c6ef74 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:12.692012745 +0000 UTC m=+10.100121627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwk2v" (UniqueName: "kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v") pod "network-check-target-sfpgt" (UID: "58e30377-aff5-40b7-918a-3f0ee9c6ef74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:09.131802 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:09.131707    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:09.132247 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:09.131856    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:09.132331 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:09.132309    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:09.132473 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:09.132449    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:10.131303 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:10.131059    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:10.131303 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:10.131201    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:11.131695 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:11.131667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:11.132154 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:11.131807    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:11.132214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:11.132195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:11.132327 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:11.132304    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:12.131279 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:12.131247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:12.131500 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.131402    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:12.622127 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:12.621575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:12.622127 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:12.621641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:12.622127 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.621767    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:12.622127 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.621803    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:12.622127 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.621842    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs podName:c5001893-317d-42d4-afc4-3392d1d442ea nodeName:}" failed. No retries permitted until 2026-07-07 15:13:20.62182398 +0000 UTC m=+18.029932862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs") pod "network-metrics-daemon-tb2xg" (UID: "c5001893-317d-42d4-afc4-3392d1d442ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:12.622127 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.621861    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret podName:247e63ad-de91-47f1-bae3-c697a7e5f702 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:20.621851296 +0000 UTC m=+18.029960184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret") pod "global-pull-secret-syncer-6njzf" (UID: "247e63ad-de91-47f1-bae3-c697a7e5f702") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:12.722907 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:12.722666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:12.722907 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.722811    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:13:12.722907 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.722832    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:13:12.722907 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.722843    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mwk2v for pod openshift-network-diagnostics/network-check-target-sfpgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:12.722907 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:12.722898    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v podName:58e30377-aff5-40b7-918a-3f0ee9c6ef74 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:20.722881099 +0000 UTC m=+18.130989993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwk2v" (UniqueName: "kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v") pod "network-check-target-sfpgt" (UID: "58e30377-aff5-40b7-918a-3f0ee9c6ef74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:13.135684 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:13.132485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:13.135684 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:13.132649    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:13.135684 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:13.133052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:13.135684 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:13.133166    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:14.131675 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:14.131645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:14.132069 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:14.131785    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:15.131593 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:15.131560    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:15.131765 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:15.131561    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:15.131765 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:15.131708    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:15.132239 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:15.131774    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:16.130968 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:16.130933    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:16.131146 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:16.131047    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:17.131201 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:17.131120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:17.131681 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:17.131281    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:17.131681 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:17.131315    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:17.131681 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:17.131367    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:18.130691 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:18.130659    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:18.130858 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:18.130767    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:19.131789 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:19.131705    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:19.131789 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:19.131729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:19.132292 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:19.131839    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:19.132292 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:19.131918    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:20.131507 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:20.131465    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:20.131773 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.131594    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:20.690737 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:20.690701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:20.691518 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:20.690748    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:20.691518 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.690859    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:20.691518 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.690861    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:20.691518 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.690921    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs podName:c5001893-317d-42d4-afc4-3392d1d442ea nodeName:}" failed. No retries permitted until 2026-07-07 15:13:36.690904545 +0000 UTC m=+34.099013434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs") pod "network-metrics-daemon-tb2xg" (UID: "c5001893-317d-42d4-afc4-3392d1d442ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:20.691518 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.690963    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret podName:247e63ad-de91-47f1-bae3-c697a7e5f702 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:36.690948936 +0000 UTC m=+34.099057819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret") pod "global-pull-secret-syncer-6njzf" (UID: "247e63ad-de91-47f1-bae3-c697a7e5f702") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:20.791616 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:20.791580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:20.791822 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.791727    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:13:20.791822 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.791743    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:13:20.791822 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.791752    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mwk2v for pod openshift-network-diagnostics/network-check-target-sfpgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:20.791822 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:20.791798    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v podName:58e30377-aff5-40b7-918a-3f0ee9c6ef74 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:36.791785236 +0000 UTC m=+34.199894120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwk2v" (UniqueName: "kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v") pod "network-check-target-sfpgt" (UID: "58e30377-aff5-40b7-918a-3f0ee9c6ef74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:21.131298 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:21.131263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:21.131511 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:21.131263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:21.131511 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:21.131443    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:21.131511 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:21.131486    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:22.131729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:22.131693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:22.132120 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:22.131810    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:23.132173 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:23.132130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:23.132650 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:23.132255    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:23.132650 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:23.132324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:23.132650 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:23.132467    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:24.131206 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:24.131173    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:24.131408 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:24.131298    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:25.131289 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.130874    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:25.132120 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.130947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:25.132120 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:25.131424    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:25.132120 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:25.131457    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:25.272949 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.272924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqm52" event={"ID":"4aeb6b5b-929c-4c7d-bcbb-2273b0102f8c","Type":"ContainerStarted","Data":"7080596de3c2913d4b50aa1d9704cfcb6da6b00e69f71c7eeaa621b04de2b3f6"}
Jul 07 15:13:25.274255 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.274234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6" event={"ID":"91a18b81-7085-4b8c-8ecf-0b3cf6d413af","Type":"ContainerStarted","Data":"d6e3f23b62ef7fa467ae1380dfb658386a8018ac9deb4e8e36710601e983c3e6"}
Jul 07 15:13:25.275414 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.275379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmfmg" event={"ID":"5d0644a9-b118-45fa-9e14-77ee2cd8c242","Type":"ContainerStarted","Data":"570decf3f6a893195db49436ef6c9f99a892b89338d4cd6e852ed1f84235bad2"}
Jul 07 15:13:25.277338 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.277320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"1bec3a123e4b05d08066c4d4fd3797bfbba2e1236bc5127313f64c560f01b60b"}
Jul 07 15:13:25.277447 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.277380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"53eee2542ca30669a24d8eae6efba7e81c2f8798db4af495872ee318f74839f2"}
Jul 07 15:13:25.277447 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.277427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"59330ac7adebcfcd62329662e2aee3be43fedfd0fc6a5a6221b3768fb04b7776"}
Jul 07 15:13:25.277571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.277471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"ea7fae4ceacb620b292b0eb572c8a4463d465b22faa0068d4bf0aaa53ac3044e"}
Jul 07 15:13:25.278621 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.278602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kslrc" event={"ID":"e95cd953-6426-481f-af3c-45c51721226f","Type":"ContainerStarted","Data":"1948579c8bb4b0fd46e162c4e01a434e6085ff493fbb43fa74b33fd436a8e469"}
Jul 07 15:13:25.279990 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.279966    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd" containerID="152e829440a964334de6e55ac8323ec54479e71192483b691d5ce772500616d0" exitCode=0
Jul 07 15:13:25.280078 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.279993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerDied","Data":"152e829440a964334de6e55ac8323ec54479e71192483b691d5ce772500616d0"}
Jul 07 15:13:25.281422 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.281381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsqlt" event={"ID":"8048ebf5-390f-44f1-9ba9-a3a7c46ad771","Type":"ContainerStarted","Data":"73dd804bf1e860aac1660f77ce9d24374b74c67fbd34f33c758a613a6f078c53"}
Jul 07 15:13:25.282614 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.282596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fjq24" event={"ID":"220a5ee3-b52f-414a-8d29-7453eda2bfb9","Type":"ContainerStarted","Data":"1b694095a2f4e79c660a340a6b78e4629c4a6e290b2de56c3b40cc5735bf4b1a"}
Jul 07 15:13:25.305907 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.305866    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-161.ec2.internal" podStartSLOduration=21.30585313 podStartE2EDuration="21.30585313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:08.253420752 +0000 UTC m=+5.661529655" watchObservedRunningTime="2026-07-07 15:13:25.30585313 +0000 UTC m=+22.713962032"
Jul 07 15:13:25.322937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.322898    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fjq24" podStartSLOduration=3.879039601 podStartE2EDuration="22.322883447s" totalImagesPullingTime="18.443843846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:25.322402727 +0000 UTC m=+22.730511625" watchObservedRunningTime="2026-07-07 15:13:25.322883447 +0000 UTC m=+22.730992350"
Jul 07 15:13:25.339664 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.339626    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hqm52" podStartSLOduration=3.880262649 podStartE2EDuration="22.339611597s" totalImagesPullingTime="18.459348948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:25.339281798 +0000 UTC m=+22.747390701" watchObservedRunningTime="2026-07-07 15:13:25.339611597 +0000 UTC m=+22.747720498"
Jul 07 15:13:25.353155 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.353117    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cmfmg" podStartSLOduration=3.915466631 podStartE2EDuration="22.353104189s" totalImagesPullingTime="18.437637558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:25.352343755 +0000 UTC m=+22.760452655" watchObservedRunningTime="2026-07-07 15:13:25.353104189 +0000 UTC m=+22.761213116"
Jul 07 15:13:25.371467 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:25.371432    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kslrc" podStartSLOduration=7.789100266 podStartE2EDuration="22.371418104s" totalImagesPullingTime="14.582317838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:25.371232892 +0000 UTC m=+22.779341795" watchObservedRunningTime="2026-07-07 15:13:25.371418104 +0000 UTC m=+22.779527008"
Jul 07 15:13:26.131120 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.131052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:26.131294 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:26.131232    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:26.212563 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.212513    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 15:13:26.286996 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.286947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzkvh" event={"ID":"1c354928-c869-4d16-afe4-d32dfdfed3a8","Type":"ContainerStarted","Data":"9ee7e0269f7c590616037263cf8a5e1ad44c66ba2f125a294872bd7fb1ba64dc"}
Jul 07 15:13:26.289036 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.288972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6" event={"ID":"91a18b81-7085-4b8c-8ecf-0b3cf6d413af","Type":"ContainerStarted","Data":"4c626a17af0dd1657513c24c4cbbf906931f682c0f32996778540aa31fcfea55"}
Jul 07 15:13:26.292494 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.292471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"0c0dc404829b523768ea4a69f1b5fffe324747892f849ddd918b7979ba119712"}
Jul 07 15:13:26.292589 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.292536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"7fb6dda089728c401d4ec27f12d50f4443ed8f11d4c13c5e0c15f7cb31df40ab"}
Jul 07 15:13:26.303132 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.303087    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rsqlt" podStartSLOduration=4.809400985 podStartE2EDuration="23.303072833s" totalImagesPullingTime="18.493671848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:25.392233473 +0000 UTC m=+22.800342357" watchObservedRunningTime="2026-07-07 15:13:26.303072833 +0000 UTC m=+23.711181736"
Jul 07 15:13:26.303290 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:26.303263    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bzkvh" podStartSLOduration=4.867183798 podStartE2EDuration="23.303251502s" totalImagesPullingTime="18.436067704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:26.302904901 +0000 UTC m=+23.711013803" watchObservedRunningTime="2026-07-07 15:13:26.303251502 +0000 UTC m=+23.711360405"
Jul 07 15:13:27.017461 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:27.017312    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T15:13:26.212540231Z","UUID":"549c574a-b206-4366-85d4-ce4b8bfa9f0f","Handler":null,"Name":"","Endpoint":""}
Jul 07 15:13:27.019236 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:27.019196    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 15:13:27.019236 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:27.019241    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 15:13:27.131022 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:27.130987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:27.131022 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:27.131012    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:27.131222 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:27.131125    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:27.131271 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:27.131241    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:28.131244 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:28.131028    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:28.131626 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:28.131297    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:28.300048 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:28.300007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6" event={"ID":"91a18b81-7085-4b8c-8ecf-0b3cf6d413af","Type":"ContainerStarted","Data":"ceaca80aa5191da1be9bab108047b4e20b9b2ef9059c3c2cb31d88c1f2103c85"}
Jul 07 15:13:28.303665 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:28.303635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"dd753c2e285889c69dc7c7afc71beb53b2d20611589fb96484cb7c75b509bfba"}
Jul 07 15:13:28.751685 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:28.751648    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:28.752281 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:28.752258    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:28.770322 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:28.770274    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cddt6" podStartSLOduration=5.012122513 podStartE2EDuration="25.770259187s" totalImagesPullingTime="20.758136674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:28.318995141 +0000 UTC m=+25.727104043" watchObservedRunningTime="2026-07-07 15:13:28.770259187 +0000 UTC m=+26.178368089"
Jul 07 15:13:29.131513 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:29.131486    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:29.132083 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:29.131490    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:29.132083 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:29.131612    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:29.132083 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:29.131728    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:30.131744 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:30.131576    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:30.132249 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:30.131821    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:30.310137 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:30.310041    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd" containerID="2ca7b940a7fdec2f4a16f43a8ba128c99cb146700f694d20ebdf092f768f95ee" exitCode=0
Jul 07 15:13:30.310281 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:30.310135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerDied","Data":"2ca7b940a7fdec2f4a16f43a8ba128c99cb146700f694d20ebdf092f768f95ee"}
Jul 07 15:13:30.314156 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:30.314129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" event={"ID":"fe6c135a-71bf-45ec-978a-697aeee6a751","Type":"ContainerStarted","Data":"6bc07302f84cfc50eb661ff7afd4276f425d6da9444b82fe10e4aba4d93ed9b8"}
Jul 07 15:13:30.314409 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:30.314368    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:30.330153 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:30.330126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:30.372996 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:30.372942    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" podStartSLOduration=8.862489012 podStartE2EDuration="27.372925282s" totalImagesPullingTime="18.51043627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:30.369981241 +0000 UTC m=+27.778090134" watchObservedRunningTime="2026-07-07 15:13:30.372925282 +0000 UTC m=+27.781034184"
Jul 07 15:13:31.130721 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:31.130689    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:31.130946 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:31.130693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:31.130946 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:31.130834    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:31.131054 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:31.130975    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:31.316158 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:31.316132    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 15:13:31.316648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:31.316557    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:31.331952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:31.331926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:32.078742 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.078549    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6njzf"]
Jul 07 15:13:32.078961 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.078823    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:32.078961 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:32.078904    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:32.081463 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.081435    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sfpgt"]
Jul 07 15:13:32.081581 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.081522    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:32.081620 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:32.081596    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:32.090900 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.090873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tb2xg"]
Jul 07 15:13:32.090998 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.090949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:32.091046 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:32.091023    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:32.320717 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.320686    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd" containerID="3f921eb8bb1472e001030ce802f6d9a73a226266be763c61f4599bfafa2d673f" exitCode=0
Jul 07 15:13:32.321081 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.320771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerDied","Data":"3f921eb8bb1472e001030ce802f6d9a73a226266be763c61f4599bfafa2d673f"}
Jul 07 15:13:32.321081 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:32.320815    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 15:13:33.322570 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:33.322539    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 15:13:33.858397 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:33.858351    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:13:34.130899 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.130809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:34.130899 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.130843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:34.130899 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.130845    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:34.131122 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:34.130912    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:34.131122 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:34.131000    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:34.131122 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:34.131072    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:34.327340 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.327308    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd" containerID="02bc10a7cde48b49c786dfabd93ca9176b99de55c9ff98b7581d7a969b256169" exitCode=0
Jul 07 15:13:34.327965 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.327413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerDied","Data":"02bc10a7cde48b49c786dfabd93ca9176b99de55c9ff98b7581d7a969b256169"}
Jul 07 15:13:34.346963 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.346920    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk" podUID="fe6c135a-71bf-45ec-978a-697aeee6a751" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 07 15:13:34.872666 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.872589    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:34.872797 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.872780    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 15:13:34.873687 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:34.873550    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kslrc"
Jul 07 15:13:36.131596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:36.131564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:36.132214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:36.131564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:36.132214 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.131686    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:36.132214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:36.131564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:36.132214 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.131767    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:36.132214 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.131849    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:36.713631 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:36.713556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:36.713631 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:36.713621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:36.713836 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.713676    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:36.713836 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.713728    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:36.713836 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.713744    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret podName:247e63ad-de91-47f1-bae3-c697a7e5f702 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:08.713726326 +0000 UTC m=+66.121835221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret") pod "global-pull-secret-syncer-6njzf" (UID: "247e63ad-de91-47f1-bae3-c697a7e5f702") : object "kube-system"/"original-pull-secret" not registered
Jul 07 15:13:36.713836 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.713762    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs podName:c5001893-317d-42d4-afc4-3392d1d442ea nodeName:}" failed. No retries permitted until 2026-07-07 15:14:08.713751169 +0000 UTC m=+66.121860050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs") pod "network-metrics-daemon-tb2xg" (UID: "c5001893-317d-42d4-afc4-3392d1d442ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 15:13:36.814094 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:36.814064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:36.814257 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.814202    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 15:13:36.814257 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.814219    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 15:13:36.814257 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.814228    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mwk2v for pod openshift-network-diagnostics/network-check-target-sfpgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:36.814375 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:36.814276    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v podName:58e30377-aff5-40b7-918a-3f0ee9c6ef74 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:08.814262553 +0000 UTC m=+66.222371434 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwk2v" (UniqueName: "kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v") pod "network-check-target-sfpgt" (UID: "58e30377-aff5-40b7-918a-3f0ee9c6ef74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 15:13:38.131677 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.131638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:38.132134 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.131685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:38.132134 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.131685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:38.132134 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:38.131765    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6njzf" podUID="247e63ad-de91-47f1-bae3-c697a7e5f702"
Jul 07 15:13:38.132134 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:38.131871    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfpgt" podUID="58e30377-aff5-40b7-918a-3f0ee9c6ef74"
Jul 07 15:13:38.132134 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:38.131961    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb2xg" podUID="c5001893-317d-42d4-afc4-3392d1d442ea"
Jul 07 15:13:38.344449 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.344421    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-161.ec2.internal" event="NodeReady"
Jul 07 15:13:38.344603 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.344554    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 15:13:38.396936 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.396836    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9jkls"]
Jul 07 15:13:38.409022 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.408995    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mtfhv"]
Jul 07 15:13:38.409155 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.409135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.412802 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.412766    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 15:13:38.413419 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.412753    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 15:13:38.413602 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.413439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pb8qd\""
Jul 07 15:13:38.425069 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.425045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mtfhv"]
Jul 07 15:13:38.425205 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.425076    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jkls"]
Jul 07 15:13:38.425205 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.425152    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.427732 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.427710    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 15:13:38.427883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.427803    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 15:13:38.427883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.427709    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-24msw\""
Jul 07 15:13:38.428110 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.428075    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 15:13:38.526085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.526054    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxkh\" (UniqueName: \"kubernetes.io/projected/04e52544-b5d7-428e-b574-a22cd624da55-kube-api-access-7gxkh\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.526085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.526096    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ea178d1-089a-4394-b4ce-fcd5ae5f8965-cert\") pod \"ingress-canary-mtfhv\" (UID: \"9ea178d1-089a-4394-b4ce-fcd5ae5f8965\") " pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.526284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.526115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqtl\" (UniqueName: \"kubernetes.io/projected/9ea178d1-089a-4394-b4ce-fcd5ae5f8965-kube-api-access-xlqtl\") pod \"ingress-canary-mtfhv\" (UID: \"9ea178d1-089a-4394-b4ce-fcd5ae5f8965\") " pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.526284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.526187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04e52544-b5d7-428e-b574-a22cd624da55-config-volume\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.526284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.526228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e52544-b5d7-428e-b574-a22cd624da55-tmp-dir\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.526284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.526268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.626954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.626912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04e52544-b5d7-428e-b574-a22cd624da55-config-volume\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.626954 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.626959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e52544-b5d7-428e-b574-a22cd624da55-tmp-dir\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.627214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.627007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.627214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.627046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxkh\" (UniqueName: \"kubernetes.io/projected/04e52544-b5d7-428e-b574-a22cd624da55-kube-api-access-7gxkh\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.627214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.627081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ea178d1-089a-4394-b4ce-fcd5ae5f8965-cert\") pod \"ingress-canary-mtfhv\" (UID: \"9ea178d1-089a-4394-b4ce-fcd5ae5f8965\") " pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.627214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.627109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqtl\" (UniqueName: \"kubernetes.io/projected/9ea178d1-089a-4394-b4ce-fcd5ae5f8965-kube-api-access-xlqtl\") pod \"ingress-canary-mtfhv\" (UID: \"9ea178d1-089a-4394-b4ce-fcd5ae5f8965\") " pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.627214 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:38.627144    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:13:38.627469 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:38.627219    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls podName:04e52544-b5d7-428e-b574-a22cd624da55 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:39.127201795 +0000 UTC m=+36.535310687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls") pod "dns-default-9jkls" (UID: "04e52544-b5d7-428e-b574-a22cd624da55") : secret "dns-default-metrics-tls" not found
Jul 07 15:13:38.627469 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.627369    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e52544-b5d7-428e-b574-a22cd624da55-tmp-dir\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.627736 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.627577    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/04e52544-b5d7-428e-b574-a22cd624da55-config-volume\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.631355 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.631339    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ea178d1-089a-4394-b4ce-fcd5ae5f8965-cert\") pod \"ingress-canary-mtfhv\" (UID: \"9ea178d1-089a-4394-b4ce-fcd5ae5f8965\") " pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.635777 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.635755    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxkh\" (UniqueName: \"kubernetes.io/projected/04e52544-b5d7-428e-b574-a22cd624da55-kube-api-access-7gxkh\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:38.635889 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.635815    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqtl\" (UniqueName: \"kubernetes.io/projected/9ea178d1-089a-4394-b4ce-fcd5ae5f8965-kube-api-access-xlqtl\") pod \"ingress-canary-mtfhv\" (UID: \"9ea178d1-089a-4394-b4ce-fcd5ae5f8965\") " pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.732997 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.732975    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mtfhv"
Jul 07 15:13:38.941195 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:38.941166    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mtfhv"]
Jul 07 15:13:38.986359 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:38.986327    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea178d1_089a_4394_b4ce_fcd5ae5f8965.slice/crio-ddf2119780a22f21693a8f179f10f264f0effe7907498d8925697a2572da0ef8 WatchSource:0}: Error finding container ddf2119780a22f21693a8f179f10f264f0effe7907498d8925697a2572da0ef8: Status 404 returned error can't find the container with id ddf2119780a22f21693a8f179f10f264f0effe7907498d8925697a2572da0ef8
Jul 07 15:13:39.130445 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:39.130405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:39.130608 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:39.130527    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:13:39.130608 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:39.130592    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls podName:04e52544-b5d7-428e-b574-a22cd624da55 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:40.130575915 +0000 UTC m=+37.538684799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls") pod "dns-default-9jkls" (UID: "04e52544-b5d7-428e-b574-a22cd624da55") : secret "dns-default-metrics-tls" not found
Jul 07 15:13:39.341541 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:39.341424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mtfhv" event={"ID":"9ea178d1-089a-4394-b4ce-fcd5ae5f8965","Type":"ContainerStarted","Data":"ddf2119780a22f21693a8f179f10f264f0effe7907498d8925697a2572da0ef8"}
Jul 07 15:13:39.344307 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:39.344285    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd" containerID="348dc9d2615cb61f98fb6eb9f82597d00b768725d108c06ecacab762cb24b5c5" exitCode=0
Jul 07 15:13:39.344378 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:39.344335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerDied","Data":"348dc9d2615cb61f98fb6eb9f82597d00b768725d108c06ecacab762cb24b5c5"}
Jul 07 15:13:40.131214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.131132    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:13:40.131214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.131179    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:13:40.131214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.131179    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:13:40.135005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.134838    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5bbhh\""
Jul 07 15:13:40.135005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.134851    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 15:13:40.135005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.134873    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 15:13:40.135005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.134924    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 15:13:40.135005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.134838    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lhgw4\""
Jul 07 15:13:40.135319 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.135066    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 15:13:40.138732 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.138708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:40.138862 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:40.138839    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:13:40.139039 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:40.138893    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls podName:04e52544-b5d7-428e-b574-a22cd624da55 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:42.138875723 +0000 UTC m=+39.546984617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls") pod "dns-default-9jkls" (UID: "04e52544-b5d7-428e-b574-a22cd624da55") : secret "dns-default-metrics-tls" not found
Jul 07 15:13:40.353192 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.353152    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd" containerID="78f558bc175a152a50bbeb59c8d3f2e4b9bc623d14cf93e9f015e2cc71de41b6" exitCode=0
Jul 07 15:13:40.353565 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:40.353227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerDied","Data":"78f558bc175a152a50bbeb59c8d3f2e4b9bc623d14cf93e9f015e2cc71de41b6"}
Jul 07 15:13:41.357550 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:41.357516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mtfhv" event={"ID":"9ea178d1-089a-4394-b4ce-fcd5ae5f8965","Type":"ContainerStarted","Data":"eab860e62c90e29a4f50f1af4a917dad2d57fb51116b8195db71ca8b9f352276"}
Jul 07 15:13:41.361156 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:41.361127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zggnx" event={"ID":"9a47aeac-5a3d-4dc2-97d3-554bfbd1c8bd","Type":"ContainerStarted","Data":"f92b6f2fd630eec582dd5810b14e08a654b8eb9a8e0de1876e442cf0c1737366"}
Jul 07 15:13:41.376558 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:41.376511    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mtfhv" podStartSLOduration=1.163563418 podStartE2EDuration="3.376496545s" totalImagesPullingTime="2.212933127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:41.375160629 +0000 UTC m=+38.783269532" watchObservedRunningTime="2026-07-07 15:13:41.376496545 +0000 UTC m=+38.784605447"
Jul 07 15:13:41.401374 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:41.401313    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zggnx" podStartSLOduration=8.866807118 podStartE2EDuration="38.401291799s" totalImagesPullingTime="29.534484681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:41.401092036 +0000 UTC m=+38.809200950" watchObservedRunningTime="2026-07-07 15:13:41.401291799 +0000 UTC m=+38.809400706"
Jul 07 15:13:42.153863 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:42.153832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:42.154046 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:42.153952    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:13:42.154046 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:42.154003    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls podName:04e52544-b5d7-428e-b574-a22cd624da55 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:46.153990114 +0000 UTC m=+43.562098994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls") pod "dns-default-9jkls" (UID: "04e52544-b5d7-428e-b574-a22cd624da55") : secret "dns-default-metrics-tls" not found
Jul 07 15:13:46.181451 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:46.181408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:46.181860 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:46.181568    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:13:46.181860 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:46.181638    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls podName:04e52544-b5d7-428e-b574-a22cd624da55 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:54.18161883 +0000 UTC m=+51.589727719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls") pod "dns-default-9jkls" (UID: "04e52544-b5d7-428e-b574-a22cd624da55") : secret "dns-default-metrics-tls" not found
Jul 07 15:13:48.088328 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.088295    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsrf"]
Jul 07 15:13:48.106223 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.106195    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsrf"]
Jul 07 15:13:48.106443 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.106302    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.108949 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.108913    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 15:13:48.109064 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.108991    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-265hp\""
Jul 07 15:13:48.109249 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.109236    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 15:13:48.109880 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.109862    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 15:13:48.109974 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.109865    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 15:13:48.194436 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.194376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7hgj\" (UniqueName: \"kubernetes.io/projected/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-api-access-t7hgj\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.194898 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.194843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-crio-socket\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.195035 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.194920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-data-volume\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.195105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.195029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.195156 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.195108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.296054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.296012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-crio-socket\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.296054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.296055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-data-volume\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.296301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.296075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.296301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.296092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.296301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.296140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7hgj\" (UniqueName: \"kubernetes.io/projected/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-api-access-t7hgj\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.296301 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.296189    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:13:48.296301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.296231    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-crio-socket\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.296301 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.296265    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:48.79624748 +0000 UTC m=+46.204356374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : configmap "kube-rbac-proxy" not found
Jul 07 15:13:48.296301 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.296280    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:48.296647 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.296349    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:48.796334326 +0000 UTC m=+46.204443205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:48.296647 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.296463    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-data-volume\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.317536 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.317513    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7hgj\" (UniqueName: \"kubernetes.io/projected/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-api-access-t7hgj\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.801641 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.801604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.801641 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:48.801641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:48.801844 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.801728    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:13:48.801844 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.801755    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:48.801844 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.801830    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:49.801817621 +0000 UTC m=+47.209926515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:48.801844 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:48.801843    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:49.801836844 +0000 UTC m=+47.209945724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : configmap "kube-rbac-proxy" not found
Jul 07 15:13:49.430560 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.430522    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"]
Jul 07 15:13:49.452158 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.452127    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.454907 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.454883    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 15:13:49.455034 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.454921    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 15:13:49.455034 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.454966    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-st7wq\""
Jul 07 15:13:49.455188 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.455176    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 15:13:49.459247 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.459225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"]
Jul 07 15:13:49.460605 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.460587    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 15:13:49.608024 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.607981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-image-registry-private-configuration\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.608204 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.608036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.608204 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.608101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-trusted-ca\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.608204 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.608132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-installation-pull-secrets\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.608204 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.608175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-bound-sa-token\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.608379 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.608214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2192f76e-09d0-40c2-934a-02c8bcf14b12-ca-trust-extracted\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.608379 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.608231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-certificates\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.608379 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.608247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjh4\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-kube-api-access-dkjh4\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709662 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.709566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-image-registry-private-configuration\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709662 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.709617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709662 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.709636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-trusted-ca\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709662 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.709660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-installation-pull-secrets\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709976 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.709681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-bound-sa-token\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709976 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.709706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2192f76e-09d0-40c2-934a-02c8bcf14b12-ca-trust-extracted\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709976 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.709730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-certificates\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.709976 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:49.709733    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:13:49.709976 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:49.709755    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd: secret "image-registry-tls" not found
Jul 07 15:13:49.709976 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:49.709816    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls podName:2192f76e-09d0-40c2-934a-02c8bcf14b12 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:50.209797476 +0000 UTC m=+47.617906374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls") pod "image-registry-7ff6ffdd6f-rf7rd" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12") : secret "image-registry-tls" not found
Jul 07 15:13:49.710269 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.710109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjh4\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-kube-api-access-dkjh4\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.710325 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.710291    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2192f76e-09d0-40c2-934a-02c8bcf14b12-ca-trust-extracted\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.710463 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.710444    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-certificates\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.710784 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.710766    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-trusted-ca\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.712138 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.712116    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-image-registry-private-configuration\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.712226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.712146    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-installation-pull-secrets\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.724296 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.724265    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-bound-sa-token\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.727621 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.727601    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjh4\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-kube-api-access-dkjh4\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:49.811246 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.811214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:49.811246 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:49.811250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:49.811480 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:49.811347    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:13:49.811480 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:49.811358    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:49.811480 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:49.811445    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:51.811428262 +0000 UTC m=+49.219537147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : configmap "kube-rbac-proxy" not found
Jul 07 15:13:49.811583 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:49.811503    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:51.811488507 +0000 UTC m=+49.219597402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:50.214362 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.214316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:50.214552 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:50.214495    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:13:50.214552 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:50.214514    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd: secret "image-registry-tls" not found
Jul 07 15:13:50.214620 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:50.214579    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls podName:2192f76e-09d0-40c2-934a-02c8bcf14b12 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:51.214560838 +0000 UTC m=+48.622669725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls") pod "image-registry-7ff6ffdd6f-rf7rd" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12") : secret "image-registry-tls" not found
Jul 07 15:13:50.526874 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.526786    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-wp6t2"]
Jul 07 15:13:50.557053 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.557024    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.560168 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.560147    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 15:13:50.560319 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.560180    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 15:13:50.560319 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.560262    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 15:13:50.561196 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.561177    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 15:13:50.561323 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.561302    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h9gdk\""
Jul 07 15:13:50.569064 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.566841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-wp6t2"]
Jul 07 15:13:50.622291 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.622264    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjq24_220a5ee3-b52f-414a-8d29-7453eda2bfb9/dns-node-resolver/0.log"
Jul 07 15:13:50.717948 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.717910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-signing-cabundle\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.718107 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.718049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-signing-key\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.718107 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.718080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh27k\" (UniqueName: \"kubernetes.io/projected/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-kube-api-access-lh27k\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.819093 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.819001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-signing-key\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.819093 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.819045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh27k\" (UniqueName: \"kubernetes.io/projected/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-kube-api-access-lh27k\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.819093 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.819074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-signing-cabundle\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.819723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.819695    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-signing-cabundle\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.821483 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.821449    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-signing-key\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.832796 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.832764    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh27k\" (UniqueName: \"kubernetes.io/projected/aa692b19-19f8-42f0-a93c-c7c8fc90ca3a-kube-api-access-lh27k\") pod \"service-ca-758f599c5-wp6t2\" (UID: \"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a\") " pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:50.866576 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:50.866539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-wp6t2"
Jul 07 15:13:51.055590 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:51.055547    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-wp6t2"]
Jul 07 15:13:51.059646 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:13:51.059617    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa692b19_19f8_42f0_a93c_c7c8fc90ca3a.slice/crio-6bc974a1099aca51a7387310b4ffc558038d8666ca728e7fe42a7a29e58b214f WatchSource:0}: Error finding container 6bc974a1099aca51a7387310b4ffc558038d8666ca728e7fe42a7a29e58b214f: Status 404 returned error can't find the container with id 6bc974a1099aca51a7387310b4ffc558038d8666ca728e7fe42a7a29e58b214f
Jul 07 15:13:51.221513 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:51.221472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:51.221699 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:51.221596    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:13:51.221699 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:51.221608    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd: secret "image-registry-tls" not found
Jul 07 15:13:51.221699 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:51.221658    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls podName:2192f76e-09d0-40c2-934a-02c8bcf14b12 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:53.221644159 +0000 UTC m=+50.629753040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls") pod "image-registry-7ff6ffdd6f-rf7rd" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12") : secret "image-registry-tls" not found
Jul 07 15:13:51.384027 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:51.383987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-wp6t2" event={"ID":"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a","Type":"ContainerStarted","Data":"6bc974a1099aca51a7387310b4ffc558038d8666ca728e7fe42a7a29e58b214f"}
Jul 07 15:13:51.438074 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:51.438044    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmfmg_5d0644a9-b118-45fa-9e14-77ee2cd8c242/node-ca/0.log"
Jul 07 15:13:51.826813 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:51.826768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:51.826813 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:51.826815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:51.827305 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:51.826899    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:13:51.827305 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:51.826968    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:55.826950792 +0000 UTC m=+53.235059677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : configmap "kube-rbac-proxy" not found
Jul 07 15:13:51.827305 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:51.826990    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:51.827305 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:51.827042    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:55.827026059 +0000 UTC m=+53.235134939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:52.626382 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:52.626353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mtfhv_9ea178d1-089a-4394-b4ce-fcd5ae5f8965/serve-healthcheck-canary/0.log"
Jul 07 15:13:53.240648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:53.240608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:53.241078 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:53.240762    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:13:53.241078 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:53.240777    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd: secret "image-registry-tls" not found
Jul 07 15:13:53.241078 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:53.240831    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls podName:2192f76e-09d0-40c2-934a-02c8bcf14b12 nodeName:}" failed. No retries permitted until 2026-07-07 15:13:57.240816204 +0000 UTC m=+54.648925089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls") pod "image-registry-7ff6ffdd6f-rf7rd" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12") : secret "image-registry-tls" not found
Jul 07 15:13:54.248732 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:54.248643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:13:54.249111 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:54.248804    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 15:13:54.249111 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:54.248874    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls podName:04e52544-b5d7-428e-b574-a22cd624da55 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:10.248857408 +0000 UTC m=+67.656966293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls") pod "dns-default-9jkls" (UID: "04e52544-b5d7-428e-b574-a22cd624da55") : secret "dns-default-metrics-tls" not found
Jul 07 15:13:54.392082 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:54.392045    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-wp6t2" event={"ID":"aa692b19-19f8-42f0-a93c-c7c8fc90ca3a","Type":"ContainerStarted","Data":"2b454d344167a7b563923dae3566c619598ceb608ae8c56f0e4ae1ad9f9e1acb"}
Jul 07 15:13:55.863336 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:55.863289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:55.863918 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:55.863354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:13:55.863918 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:55.863437    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 15:13:55.863918 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:55.863515    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:03.863495467 +0000 UTC m=+61.271604360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : configmap "kube-rbac-proxy" not found
Jul 07 15:13:55.863918 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:55.863593    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:55.863918 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:55.863660    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:03.863645908 +0000 UTC m=+61.271754803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:13:57.274694 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:13:57.274656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:13:57.275155 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:57.274850    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 15:13:57.275155 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:57.274878    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd: secret "image-registry-tls" not found
Jul 07 15:13:57.275155 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:13:57.274948    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls podName:2192f76e-09d0-40c2-934a-02c8bcf14b12 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:05.274927438 +0000 UTC m=+62.683036335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls") pod "image-registry-7ff6ffdd6f-rf7rd" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12") : secret "image-registry-tls" not found
Jul 07 15:14:03.926761 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:03.926719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:14:03.926761 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:03.926771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:14:03.927427 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:03.926882    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 15:14:03.927427 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:03.926937    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls podName:0d8f6c7c-257b-4bf4-adef-ca757ed4d908 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:19.926923132 +0000 UTC m=+77.335032012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmsrf" (UID: "0d8f6c7c-257b-4bf4-adef-ca757ed4d908") : secret "insights-runtime-extractor-tls" not found
Jul 07 15:14:03.927427 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:03.927248    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:14:04.337876 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:04.337807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2jk"
Jul 07 15:14:04.369781 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:04.369736    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-wp6t2" podStartSLOduration=11.514113067 podStartE2EDuration="14.369720059s" totalImagesPullingTime="2.855606992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:54.412790583 +0000 UTC m=+51.820899489" watchObservedRunningTime="2026-07-07 15:14:04.369720059 +0000 UTC m=+61.777828960"
Jul 07 15:14:05.335973 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:05.335937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:14:05.338276 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:05.338241    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"image-registry-7ff6ffdd6f-rf7rd\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") " pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:14:05.365486 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:05.365461    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-st7wq\""
Jul 07 15:14:05.373168 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:05.373146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:14:05.521019 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:05.520987    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"]
Jul 07 15:14:05.524443 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:05.524380    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2192f76e_09d0_40c2_934a_02c8bcf14b12.slice/crio-e64ccadc62591bb63900c57d232ffa385f082b8ff36724186a6a094c5cbb840a WatchSource:0}: Error finding container e64ccadc62591bb63900c57d232ffa385f082b8ff36724186a6a094c5cbb840a: Status 404 returned error can't find the container with id e64ccadc62591bb63900c57d232ffa385f082b8ff36724186a6a094c5cbb840a
Jul 07 15:14:06.423991 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:06.423958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd" event={"ID":"2192f76e-09d0-40c2-934a-02c8bcf14b12","Type":"ContainerStarted","Data":"5da8b92d1f32d3bde03b870ce4c61b1dcc4e96b802f272700dfc06fc042e0200"}
Jul 07 15:14:06.424367 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:06.424005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd" event={"ID":"2192f76e-09d0-40c2-934a-02c8bcf14b12","Type":"ContainerStarted","Data":"e64ccadc62591bb63900c57d232ffa385f082b8ff36724186a6a094c5cbb840a"}
Jul 07 15:14:06.424367 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:06.424088    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:14:06.450935 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:06.450886    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd" podStartSLOduration=17.450869012 podStartE2EDuration="17.450869012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:06.449807402 +0000 UTC m=+63.857916318" watchObservedRunningTime="2026-07-07 15:14:06.450869012 +0000 UTC m=+63.858977912"
Jul 07 15:14:08.762459 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.762426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:14:08.762877 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.762480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:14:08.766324 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.766304    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 15:14:08.767067 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.767052    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 15:14:08.775199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.775174    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5001893-317d-42d4-afc4-3392d1d442ea-metrics-certs\") pod \"network-metrics-daemon-tb2xg\" (UID: \"c5001893-317d-42d4-afc4-3392d1d442ea\") " pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:14:08.775199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.775197    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247e63ad-de91-47f1-bae3-c697a7e5f702-original-pull-secret\") pod \"global-pull-secret-syncer-6njzf\" (UID: \"247e63ad-de91-47f1-bae3-c697a7e5f702\") " pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:14:08.863353 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.863318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:14:08.867099 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.867071    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 15:14:08.876381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.876354    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 15:14:08.886679 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.886637    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwk2v\" (UniqueName: \"kubernetes.io/projected/58e30377-aff5-40b7-918a-3f0ee9c6ef74-kube-api-access-mwk2v\") pod \"network-check-target-sfpgt\" (UID: \"58e30377-aff5-40b7-918a-3f0ee9c6ef74\") " pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:14:08.944964 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.944928    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5bbhh\""
Jul 07 15:14:08.952309 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.952288    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lhgw4\""
Jul 07 15:14:08.953340 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.953323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:14:08.955314 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.955298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6njzf"
Jul 07 15:14:08.960967 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:08.960948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb2xg"
Jul 07 15:14:09.100845 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:09.100115    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sfpgt"]
Jul 07 15:14:09.102566 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:09.102529    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e30377_aff5_40b7_918a_3f0ee9c6ef74.slice/crio-d99c1e042b901809b6473bcd840e1a0554c270378e01cef4b9f0433f451c30a3 WatchSource:0}: Error finding container d99c1e042b901809b6473bcd840e1a0554c270378e01cef4b9f0433f451c30a3: Status 404 returned error can't find the container with id d99c1e042b901809b6473bcd840e1a0554c270378e01cef4b9f0433f451c30a3
Jul 07 15:14:09.114649 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:09.114623    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6njzf"]
Jul 07 15:14:09.116923 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:09.116897    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247e63ad_de91_47f1_bae3_c697a7e5f702.slice/crio-6bcb3b44480daf991dc48951989342ccca006ec8496d4c084d4dc349d8d2b197 WatchSource:0}: Error finding container 6bcb3b44480daf991dc48951989342ccca006ec8496d4c084d4dc349d8d2b197: Status 404 returned error can't find the container with id 6bcb3b44480daf991dc48951989342ccca006ec8496d4c084d4dc349d8d2b197
Jul 07 15:14:09.134606 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:09.134582    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tb2xg"]
Jul 07 15:14:09.137081 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:09.137058    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5001893_317d_42d4_afc4_3392d1d442ea.slice/crio-3011a111877be43d20dc292d870b91d226d5632d85886b747bfdcddaa258af98 WatchSource:0}: Error finding container 3011a111877be43d20dc292d870b91d226d5632d85886b747bfdcddaa258af98: Status 404 returned error can't find the container with id 3011a111877be43d20dc292d870b91d226d5632d85886b747bfdcddaa258af98
Jul 07 15:14:09.432860 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:09.432826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb2xg" event={"ID":"c5001893-317d-42d4-afc4-3392d1d442ea","Type":"ContainerStarted","Data":"3011a111877be43d20dc292d870b91d226d5632d85886b747bfdcddaa258af98"}
Jul 07 15:14:09.433845 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:09.433818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6njzf" event={"ID":"247e63ad-de91-47f1-bae3-c697a7e5f702","Type":"ContainerStarted","Data":"6bcb3b44480daf991dc48951989342ccca006ec8496d4c084d4dc349d8d2b197"}
Jul 07 15:14:09.434850 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:09.434829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sfpgt" event={"ID":"58e30377-aff5-40b7-918a-3f0ee9c6ef74","Type":"ContainerStarted","Data":"d99c1e042b901809b6473bcd840e1a0554c270378e01cef4b9f0433f451c30a3"}
Jul 07 15:14:10.273955 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:10.273915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:14:10.277273 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:10.277214    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/04e52544-b5d7-428e-b574-a22cd624da55-metrics-tls\") pod \"dns-default-9jkls\" (UID: \"04e52544-b5d7-428e-b574-a22cd624da55\") " pod="openshift-dns/dns-default-9jkls"
Jul 07 15:14:10.441511 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:10.441469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb2xg" event={"ID":"c5001893-317d-42d4-afc4-3392d1d442ea","Type":"ContainerStarted","Data":"1c4a448116800c09e831b5b9a37b9f5ddb967329162585548e3d1ea02e14d008"}
Jul 07 15:14:10.525714 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:10.525631    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pb8qd\""
Jul 07 15:14:10.533367 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:10.533336    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jkls"
Jul 07 15:14:10.692174 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:10.692140    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jkls"]
Jul 07 15:14:10.695487 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:10.695458    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e52544_b5d7_428e_b574_a22cd624da55.slice/crio-0e254304c0a17330e721c9863a2e74bd6c6666fa73f6fdc603494e385da2478a WatchSource:0}: Error finding container 0e254304c0a17330e721c9863a2e74bd6c6666fa73f6fdc603494e385da2478a: Status 404 returned error can't find the container with id 0e254304c0a17330e721c9863a2e74bd6c6666fa73f6fdc603494e385da2478a
Jul 07 15:14:11.448424 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:11.448183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb2xg" event={"ID":"c5001893-317d-42d4-afc4-3392d1d442ea","Type":"ContainerStarted","Data":"c2ab30d81975a74b55f933e5b019d4eb716b23cc7c42ea08e113c57a5a2a751e"}
Jul 07 15:14:11.450596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:11.450532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jkls" event={"ID":"04e52544-b5d7-428e-b574-a22cd624da55","Type":"ContainerStarted","Data":"0e254304c0a17330e721c9863a2e74bd6c6666fa73f6fdc603494e385da2478a"}
Jul 07 15:14:11.467321 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:11.467263    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tb2xg" podStartSLOduration=67.376261532 podStartE2EDuration="1m8.467240463s" totalImagesPullingTime="1.090978931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:11.465884404 +0000 UTC m=+68.873993299" watchObservedRunningTime="2026-07-07 15:14:11.467240463 +0000 UTC m=+68.875349366"
Jul 07 15:14:13.815584 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.815540    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"]
Jul 07 15:14:13.835244 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.835217    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"]
Jul 07 15:14:13.835406 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.835340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:13.837984 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.837959    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 15:14:13.838117 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.837967    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 15:14:13.838294 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.838279    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 15:14:13.839070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.839051    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 15:14:13.865767 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.865736    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"]
Jul 07 15:14:13.865906 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.865777    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"]
Jul 07 15:14:13.865906 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.865858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:13.869299 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.869275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q67kb\""
Jul 07 15:14:13.869651 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.869635    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 15:14:13.903060 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.903030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4mn\" (UniqueName: \"kubernetes.io/projected/f8955405-99c5-43b1-8b96-c4c626015076-kube-api-access-zz4mn\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:13.903216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.903078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8955405-99c5-43b1-8b96-c4c626015076-tmp\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:13.903216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.903100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqmc\" (UniqueName: \"kubernetes.io/projected/73c08e78-fddb-4051-bd4d-10f21d586ebe-kube-api-access-5sqmc\") pod \"managed-serviceaccount-addon-agent-5df764648-glvfk\" (UID: \"73c08e78-fddb-4051-bd4d-10f21d586ebe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:13.903216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.903165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f8955405-99c5-43b1-8b96-c4c626015076-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:13.903216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.903213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73c08e78-fddb-4051-bd4d-10f21d586ebe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df764648-glvfk\" (UID: \"73c08e78-fddb-4051-bd4d-10f21d586ebe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:13.906481 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:13.906459    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"]
Jul 07 15:14:14.004614 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.004577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8955405-99c5-43b1-8b96-c4c626015076-tmp\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:14.004614 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.004613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqmc\" (UniqueName: \"kubernetes.io/projected/73c08e78-fddb-4051-bd4d-10f21d586ebe-kube-api-access-5sqmc\") pod \"managed-serviceaccount-addon-agent-5df764648-glvfk\" (UID: \"73c08e78-fddb-4051-bd4d-10f21d586ebe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:14.004814 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.004632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f8955405-99c5-43b1-8b96-c4c626015076-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:14.004814 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.004672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73c08e78-fddb-4051-bd4d-10f21d586ebe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df764648-glvfk\" (UID: \"73c08e78-fddb-4051-bd4d-10f21d586ebe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:14.004881 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.004856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4mn\" (UniqueName: \"kubernetes.io/projected/f8955405-99c5-43b1-8b96-c4c626015076-kube-api-access-zz4mn\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:14.007121 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.007098    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73c08e78-fddb-4051-bd4d-10f21d586ebe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5df764648-glvfk\" (UID: \"73c08e78-fddb-4051-bd4d-10f21d586ebe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:14.015101 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.015073    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8955405-99c5-43b1-8b96-c4c626015076-tmp\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:14.015377 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.015361    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f8955405-99c5-43b1-8b96-c4c626015076-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:14.038941 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.038904    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqmc\" (UniqueName: \"kubernetes.io/projected/73c08e78-fddb-4051-bd4d-10f21d586ebe-kube-api-access-5sqmc\") pod \"managed-serviceaccount-addon-agent-5df764648-glvfk\" (UID: \"73c08e78-fddb-4051-bd4d-10f21d586ebe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:14.041963 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.041932    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4mn\" (UniqueName: \"kubernetes.io/projected/f8955405-99c5-43b1-8b96-c4c626015076-kube-api-access-zz4mn\") pod \"klusterlet-addon-workmgr-84f48dbd5-tjk5l\" (UID: \"f8955405-99c5-43b1-8b96-c4c626015076\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:14.145460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.145423    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:14.189301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.189262    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"
Jul 07 15:14:14.947856 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.947799    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk"]
Jul 07 15:14:14.952827 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:14.952772    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73c08e78_fddb_4051_bd4d_10f21d586ebe.slice/crio-9c8bbc224cb1bda8ed506facb75eebd18ad530bc234d1f2d7a1d4bd0c4ab4435 WatchSource:0}: Error finding container 9c8bbc224cb1bda8ed506facb75eebd18ad530bc234d1f2d7a1d4bd0c4ab4435: Status 404 returned error can't find the container with id 9c8bbc224cb1bda8ed506facb75eebd18ad530bc234d1f2d7a1d4bd0c4ab4435
Jul 07 15:14:14.968495 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:14.968425    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"]
Jul 07 15:14:14.972581 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:14.972544    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8955405_99c5_43b1_8b96_c4c626015076.slice/crio-9c6a8c8884a246e7bc1c43546867834c41a2b03b40c4b5a049955ff24caada16 WatchSource:0}: Error finding container 9c6a8c8884a246e7bc1c43546867834c41a2b03b40c4b5a049955ff24caada16: Status 404 returned error can't find the container with id 9c6a8c8884a246e7bc1c43546867834c41a2b03b40c4b5a049955ff24caada16
Jul 07 15:14:15.468679 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.468639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk" event={"ID":"73c08e78-fddb-4051-bd4d-10f21d586ebe","Type":"ContainerStarted","Data":"9c8bbc224cb1bda8ed506facb75eebd18ad530bc234d1f2d7a1d4bd0c4ab4435"}
Jul 07 15:14:15.470374 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.470349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jkls" event={"ID":"04e52544-b5d7-428e-b574-a22cd624da55","Type":"ContainerStarted","Data":"bd12924444208bf1f81f8422d1d3a967f9678b96033c8982dcf2bb77c3ed8665"}
Jul 07 15:14:15.470526 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.470440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jkls" event={"ID":"04e52544-b5d7-428e-b574-a22cd624da55","Type":"ContainerStarted","Data":"00f6346aa38eea7d5611e063c949214fb0981fb438301b422d4befcf9e8f0e78"}
Jul 07 15:14:15.470526 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.470471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9jkls"
Jul 07 15:14:15.471766 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.471735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6njzf" event={"ID":"247e63ad-de91-47f1-bae3-c697a7e5f702","Type":"ContainerStarted","Data":"31eb22504ecf900e9c28749ec8e106f326ddbf75bcdfc5f4bdc79ab20ea2a8b2"}
Jul 07 15:14:15.472813 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.472790    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l" event={"ID":"f8955405-99c5-43b1-8b96-c4c626015076","Type":"ContainerStarted","Data":"9c6a8c8884a246e7bc1c43546867834c41a2b03b40c4b5a049955ff24caada16"}
Jul 07 15:14:15.474272 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.474250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sfpgt" event={"ID":"58e30377-aff5-40b7-918a-3f0ee9c6ef74","Type":"ContainerStarted","Data":"60ecf9395e397c613daadc67a5e6694307bc131fc39b3dd6b56ddc9f3752c014"}
Jul 07 15:14:15.474409 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.474372    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:14:15.489921 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.489875    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9jkls" podStartSLOduration=33.396661883 podStartE2EDuration="37.48986091s" totalImagesPullingTime="4.093199027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:15.488323978 +0000 UTC m=+72.896432909" watchObservedRunningTime="2026-07-07 15:14:15.48986091 +0000 UTC m=+72.897969856"
Jul 07 15:14:15.506488 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.506439    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sfpgt" podStartSLOduration=66.820010736 podStartE2EDuration="1m12.506420429s" totalImagesPullingTime="5.686409693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:15.5055293 +0000 UTC m=+72.913638201" watchObservedRunningTime="2026-07-07 15:14:15.506420429 +0000 UTC m=+72.914529330"
Jul 07 15:14:15.524824 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:15.524776    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6njzf" podStartSLOduration=66.844030869 podStartE2EDuration="1m12.524761685s" totalImagesPullingTime="5.680730816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:15.524297359 +0000 UTC m=+72.932406263" watchObservedRunningTime="2026-07-07 15:14:15.524761685 +0000 UTC m=+72.932870565"
Jul 07 15:14:18.489075 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.489030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk" event={"ID":"73c08e78-fddb-4051-bd4d-10f21d586ebe","Type":"ContainerStarted","Data":"3b885ae59e075940cb7c0ae49acc269408996ee0202bf1f0a01433542c1fab0b"}
Jul 07 15:14:18.511741 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.511673    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5df764648-glvfk" podStartSLOduration=2.583099712 podStartE2EDuration="5.511650796s" totalImagesPullingTime="2.928551084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:18.510937097 +0000 UTC m=+75.919046026" watchObservedRunningTime="2026-07-07 15:14:18.511650796 +0000 UTC m=+75.919759699"
Jul 07 15:14:18.879167 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.879096    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-9xbsw"]
Jul 07 15:14:18.883522 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.882625    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9xbsw"
Jul 07 15:14:18.894853 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.894836    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4t7ns\""
Jul 07 15:14:18.895173 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.895157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 15:14:18.895622 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.895608    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 15:14:18.906349 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.906316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9xbsw"]
Jul 07 15:14:18.957749 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:18.957712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2xv\" (UniqueName: \"kubernetes.io/projected/80bb84da-c2ad-47db-a15f-1df528b69cb0-kube-api-access-rd2xv\") pod \"downloads-685bfb5f64-9xbsw\" (UID: \"80bb84da-c2ad-47db-a15f-1df528b69cb0\") " pod="openshift-console/downloads-685bfb5f64-9xbsw"
Jul 07 15:14:19.058427 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.058374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2xv\" (UniqueName: \"kubernetes.io/projected/80bb84da-c2ad-47db-a15f-1df528b69cb0-kube-api-access-rd2xv\") pod \"downloads-685bfb5f64-9xbsw\" (UID: \"80bb84da-c2ad-47db-a15f-1df528b69cb0\") " pod="openshift-console/downloads-685bfb5f64-9xbsw"
Jul 07 15:14:19.070505 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.070480    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2xv\" (UniqueName: \"kubernetes.io/projected/80bb84da-c2ad-47db-a15f-1df528b69cb0-kube-api-access-rd2xv\") pod \"downloads-685bfb5f64-9xbsw\" (UID: \"80bb84da-c2ad-47db-a15f-1df528b69cb0\") " pod="openshift-console/downloads-685bfb5f64-9xbsw"
Jul 07 15:14:19.193004 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.192976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9xbsw"
Jul 07 15:14:19.326919 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.326882    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9xbsw"]
Jul 07 15:14:19.330543 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:19.330516    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bb84da_c2ad_47db_a15f_1df528b69cb0.slice/crio-7728e9cfe90091c94c5a0c608d343cf289e9400d4294e0a89a15c8a7729f4360 WatchSource:0}: Error finding container 7728e9cfe90091c94c5a0c608d343cf289e9400d4294e0a89a15c8a7729f4360: Status 404 returned error can't find the container with id 7728e9cfe90091c94c5a0c608d343cf289e9400d4294e0a89a15c8a7729f4360
Jul 07 15:14:19.492986 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.492886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9xbsw" event={"ID":"80bb84da-c2ad-47db-a15f-1df528b69cb0","Type":"ContainerStarted","Data":"7728e9cfe90091c94c5a0c608d343cf289e9400d4294e0a89a15c8a7729f4360"}
Jul 07 15:14:19.494298 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.494262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l" event={"ID":"f8955405-99c5-43b1-8b96-c4c626015076","Type":"ContainerStarted","Data":"d292d1ddba6b0eee24bd9b5b263de10d63528fabbee7402e1fbc8b9c0a09bb0d"}
Jul 07 15:14:19.519021 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.518970    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l" podStartSLOduration=2.389506461 podStartE2EDuration="6.51895056s" totalImagesPullingTime="4.129444099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:19.517729372 +0000 UTC m=+76.925838285" watchObservedRunningTime="2026-07-07 15:14:19.51895056 +0000 UTC m=+76.927059463"
Jul 07 15:14:19.964841 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.964799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:14:19.967467 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:19.967435    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d8f6c7c-257b-4bf4-adef-ca757ed4d908-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsrf\" (UID: \"0d8f6c7c-257b-4bf4-adef-ca757ed4d908\") " pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:14:20.064713 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.064680    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"]
Jul 07 15:14:20.066942 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.066921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.070608 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.070580    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 15:14:20.070608 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.070601    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 15:14:20.071664 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.071616    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 15:14:20.071664 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.071614    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 15:14:20.071664 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.071664    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 15:14:20.071869 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.071669    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q8x94\""
Jul 07 15:14:20.085177 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.085156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"]
Jul 07 15:14:20.166674 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.166634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.166865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.166684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.166865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.166711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9pcq\" (UniqueName: \"kubernetes.io/projected/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-kube-api-access-w9pcq\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.166865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.166777    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.218496 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.218384    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-265hp\""
Jul 07 15:14:20.226461 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.226436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmsrf"
Jul 07 15:14:20.267306 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.267202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.267306 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.267281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.267561 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.267313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.267561 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.267340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9pcq\" (UniqueName: \"kubernetes.io/projected/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-kube-api-access-w9pcq\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.267561 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:20.267373    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 07 15:14:20.267561 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:20.267470    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-tls podName:c5e1b6d5-0e85-4978-a3ab-21ed4158c03e nodeName:}" failed. No retries permitted until 2026-07-07 15:14:20.767447035 +0000 UTC m=+78.175555932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-clzvl" (UID: "c5e1b6d5-0e85-4978-a3ab-21ed4158c03e") : secret "prometheus-operator-tls" not found
Jul 07 15:14:20.268286 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.268257    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.270129 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.270102    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.279508 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.279446    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9pcq\" (UniqueName: \"kubernetes.io/projected/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-kube-api-access-w9pcq\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.392135 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.392104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsrf"]
Jul 07 15:14:20.395529 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:20.395499    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8f6c7c_257b_4bf4_adef_ca757ed4d908.slice/crio-e2bc1e356b94681b377c0c4dc9fcf48f77e7ef40318eca45f88401cecc667f1a WatchSource:0}: Error finding container e2bc1e356b94681b377c0c4dc9fcf48f77e7ef40318eca45f88401cecc667f1a: Status 404 returned error can't find the container with id e2bc1e356b94681b377c0c4dc9fcf48f77e7ef40318eca45f88401cecc667f1a
Jul 07 15:14:20.500421 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.500308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsrf" event={"ID":"0d8f6c7c-257b-4bf4-adef-ca757ed4d908","Type":"ContainerStarted","Data":"e2bc1e356b94681b377c0c4dc9fcf48f77e7ef40318eca45f88401cecc667f1a"}
Jul 07 15:14:20.500853 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.500553    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:20.502501 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.502478    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f48dbd5-tjk5l"
Jul 07 15:14:20.771381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.771282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.773942 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.773917    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5e1b6d5-0e85-4978-a3ab-21ed4158c03e-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-clzvl\" (UID: \"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:20.983381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:20.983147    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"
Jul 07 15:14:21.175031 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:21.174951    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl"]
Jul 07 15:14:21.179098 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:21.179063    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e1b6d5_0e85_4978_a3ab_21ed4158c03e.slice/crio-552075c69520ad3f566f2c61370d4de7ec2a497fb5c894a32b83040e7eacf579 WatchSource:0}: Error finding container 552075c69520ad3f566f2c61370d4de7ec2a497fb5c894a32b83040e7eacf579: Status 404 returned error can't find the container with id 552075c69520ad3f566f2c61370d4de7ec2a497fb5c894a32b83040e7eacf579
Jul 07 15:14:21.506474 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:21.506439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl" event={"ID":"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e","Type":"ContainerStarted","Data":"552075c69520ad3f566f2c61370d4de7ec2a497fb5c894a32b83040e7eacf579"}
Jul 07 15:14:23.914794 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:23.914753    2560 patch_prober.go:28] interesting pod/image-registry-7ff6ffdd6f-rf7rd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 15:14:23.915252 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:23.914820    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd" podUID="2192f76e-09d0-40c2-934a-02c8bcf14b12" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:14:25.483458 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:25.483415    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9jkls"
Jul 07 15:14:25.525269 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:25.525215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsrf" event={"ID":"0d8f6c7c-257b-4bf4-adef-ca757ed4d908","Type":"ContainerStarted","Data":"7661cc6f8e5e726076a19f68e7c673851f386bd97e922d22db7aa5d07f916089"}
Jul 07 15:14:25.527520 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:25.527425    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl" event={"ID":"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e","Type":"ContainerStarted","Data":"f9d28f36e5c0426a7a05d48bb3b8f165de9684183132a5ff2918bb3110fb3540"}
Jul 07 15:14:25.527520 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:25.527486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl" event={"ID":"c5e1b6d5-0e85-4978-a3ab-21ed4158c03e","Type":"ContainerStarted","Data":"cedf854087c797bbf6c9945829dd93b547f7a127e07119f3d4ed03423e32ad38"}
Jul 07 15:14:25.547211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:25.547134    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-clzvl" podStartSLOduration=1.8399918450000001 podStartE2EDuration="5.547110978s" totalImagesPullingTime="3.707119133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:25.546997385 +0000 UTC m=+82.955106287" watchObservedRunningTime="2026-07-07 15:14:25.547110978 +0000 UTC m=+82.955219881"
Jul 07 15:14:26.534455 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:26.534411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsrf" event={"ID":"0d8f6c7c-257b-4bf4-adef-ca757ed4d908","Type":"ContainerStarted","Data":"33646c2d2cca0cd826f6015ef223fb418f6cbf1c254c751178d934c860947e35"}
Jul 07 15:14:27.485710 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.485358    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"]
Jul 07 15:14:27.495279 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.495254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.503624 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.502885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"]
Jul 07 15:14:27.503624 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.503019    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 15:14:27.503624 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.503242    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 15:14:27.503624 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.503478    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xg4z6\""
Jul 07 15:14:27.504108 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.503956    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 15:14:27.517602 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.517582    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nqx8f"]
Jul 07 15:14:27.520203 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.520186    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.524470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.524453    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 15:14:27.524859 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.524845    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fzzpb\""
Jul 07 15:14:27.525085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.524869    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 15:14:27.525305 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.524917    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20b906c3-d257-41e0-bdeb-a28002532276-metrics-client-ca\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-wtmp\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639292    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-tls\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-accelerators-collector-config\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-root\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-sys\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639466    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-textfile\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sz58\" (UniqueName: \"kubernetes.io/projected/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-api-access-9sz58\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w96g\" (UniqueName: \"kubernetes.io/projected/20b906c3-d257-41e0-bdeb-a28002532276-kube-api-access-8w96g\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.639809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.639658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-sys\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-textfile\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sz58\" (UniqueName: \"kubernetes.io/projected/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-api-access-9sz58\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w96g\" (UniqueName: \"kubernetes.io/projected/20b906c3-d257-41e0-bdeb-a28002532276-kube-api-access-8w96g\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.740998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20b906c3-d257-41e0-bdeb-a28002532276-metrics-client-ca\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-wtmp\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-tls\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-accelerators-collector-config\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-root\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.742470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741338    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-root\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.743504 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741383    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-sys\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.743504 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.741690    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-textfile\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.743504 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.742322    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.743504 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.742872    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-wtmp\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.743700 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:27.743557    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 15:14:27.743700 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:27.743620    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-tls podName:20b906c3-d257-41e0-bdeb-a28002532276 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:28.243598511 +0000 UTC m=+85.651707398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-tls") pod "node-exporter-nqx8f" (UID: "20b906c3-d257-41e0-bdeb-a28002532276") : secret "node-exporter-tls" not found
Jul 07 15:14:27.743900 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:27.743879    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 07 15:14:27.743953 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:14:27.743940    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-tls podName:dbd0fb21-7b50-4317-bc2c-e9a903f633f1 nodeName:}" failed. No retries permitted until 2026-07-07 15:14:28.243924862 +0000 UTC m=+85.652033742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-hdf2t" (UID: "dbd0fb21-7b50-4317-bc2c-e9a903f633f1") : secret "kube-state-metrics-tls" not found
Jul 07 15:14:27.745369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.744767    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.745369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.745102    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20b906c3-d257-41e0-bdeb-a28002532276-metrics-client-ca\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.745369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.745198    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-accelerators-collector-config\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.745655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.745380    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.747805 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.747747    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.759096 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.750200    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:27.769850 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.769802    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w96g\" (UniqueName: \"kubernetes.io/projected/20b906c3-d257-41e0-bdeb-a28002532276-kube-api-access-8w96g\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:27.770048 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:27.770021    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sz58\" (UniqueName: \"kubernetes.io/projected/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-api-access-9sz58\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:28.247754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:28.247711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-tls\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:28.247939 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:28.247825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:28.250822 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:28.250768    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20b906c3-d257-41e0-bdeb-a28002532276-node-exporter-tls\") pod \"node-exporter-nqx8f\" (UID: \"20b906c3-d257-41e0-bdeb-a28002532276\") " pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:28.251128 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:28.251099    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbd0fb21-7b50-4317-bc2c-e9a903f633f1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-hdf2t\" (UID: \"dbd0fb21-7b50-4317-bc2c-e9a903f633f1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:28.418379 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:28.417768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"
Jul 07 15:14:28.431811 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:28.431784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nqx8f"
Jul 07 15:14:28.587196 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:28.587100    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b906c3_d257_41e0_bdeb_a28002532276.slice/crio-544a90e2180bbe559b90e11dc6c23d8d53cf8d267586bb47e6d213f5b5340b24 WatchSource:0}: Error finding container 544a90e2180bbe559b90e11dc6c23d8d53cf8d267586bb47e6d213f5b5340b24: Status 404 returned error can't find the container with id 544a90e2180bbe559b90e11dc6c23d8d53cf8d267586bb47e6d213f5b5340b24
Jul 07 15:14:28.729011 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:28.728954    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t"]
Jul 07 15:14:28.732750 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:28.732721    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd0fb21_7b50_4317_bc2c_e9a903f633f1.slice/crio-dba079c71629465b512424077551d1618495ee372001c2b86f0fd893d69193b5 WatchSource:0}: Error finding container dba079c71629465b512424077551d1618495ee372001c2b86f0fd893d69193b5: Status 404 returned error can't find the container with id dba079c71629465b512424077551d1618495ee372001c2b86f0fd893d69193b5
Jul 07 15:14:29.550873 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.550836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsrf" event={"ID":"0d8f6c7c-257b-4bf4-adef-ca757ed4d908","Type":"ContainerStarted","Data":"fc5e9f5c3e68fa107c03b474b862e747cf25c5fc58c12800f9bd19e090bfd001"}
Jul 07 15:14:29.553560 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.553523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqx8f" event={"ID":"20b906c3-d257-41e0-bdeb-a28002532276","Type":"ContainerStarted","Data":"544a90e2180bbe559b90e11dc6c23d8d53cf8d267586bb47e6d213f5b5340b24"}
Jul 07 15:14:29.555057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.555032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t" event={"ID":"dbd0fb21-7b50-4317-bc2c-e9a903f633f1","Type":"ContainerStarted","Data":"dba079c71629465b512424077551d1618495ee372001c2b86f0fd893d69193b5"}
Jul 07 15:14:29.578915 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.578550    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xmsrf" podStartSLOduration=33.513120548 podStartE2EDuration="41.578529708s" totalImagesPullingTime="8.06540916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:13:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:29.57804062 +0000 UTC m=+86.986149522" watchObservedRunningTime="2026-07-07 15:14:29.578529708 +0000 UTC m=+86.986638612"
Jul 07 15:14:29.707774 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.707743    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556bd58ff-97kq7"]
Jul 07 15:14:29.711067 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.711046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.713958 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.713748    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nqx2c\""
Jul 07 15:14:29.713958 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.713819    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 15:14:29.713958 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.713836    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 15:14:29.713958 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.713755    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 15:14:29.714225 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.714048    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 15:14:29.714225 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.714092    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 15:14:29.725507 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.725447    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556bd58ff-97kq7"]
Jul 07 15:14:29.864071 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.863944    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-serving-cert\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.864071 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.864042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nwj\" (UniqueName: \"kubernetes.io/projected/a6ac93f3-5480-48a3-8438-101ebf186d86-kube-api-access-l7nwj\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.864491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.864115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-console-config\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.864491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.864422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-oauth-config\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.864491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.864456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-oauth-serving-cert\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.864608 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.864539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-service-ca\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.965642 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.965603    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-serving-cert\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.965830 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.965660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nwj\" (UniqueName: \"kubernetes.io/projected/a6ac93f3-5480-48a3-8438-101ebf186d86-kube-api-access-l7nwj\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.965830 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.965707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-console-config\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.965830 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.965744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-oauth-config\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.965830 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.965771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-oauth-serving-cert\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.965830 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.965802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-service-ca\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.966617 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.966551    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-oauth-serving-cert\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.966617 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.966598    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-service-ca\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.966805 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.966606    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-console-config\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.968630 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.968607    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-serving-cert\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.968804 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.968782    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-oauth-config\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:29.976087 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:29.976025    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nwj\" (UniqueName: \"kubernetes.io/projected/a6ac93f3-5480-48a3-8438-101ebf186d86-kube-api-access-l7nwj\") pod \"console-556bd58ff-97kq7\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") " pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:30.025848 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:30.025769    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:30.444300 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:30.444263    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556bd58ff-97kq7"]
Jul 07 15:14:30.565793 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:30.565753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556bd58ff-97kq7" event={"ID":"a6ac93f3-5480-48a3-8438-101ebf186d86","Type":"ContainerStarted","Data":"f7a478bc95515985931bbe9a19249a2af604745f71b0eb5dbdaa72695a4eaafd"}
Jul 07 15:14:30.567894 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:30.567804    2560 generic.go:356] "Generic (PLEG): container finished" podID="20b906c3-d257-41e0-bdeb-a28002532276" containerID="7b1102f976d79bceec1dd00b09eb03875be15cc24a548c89575bd115357ff860" exitCode=0
Jul 07 15:14:30.568124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:30.568056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqx8f" event={"ID":"20b906c3-d257-41e0-bdeb-a28002532276","Type":"ContainerDied","Data":"7b1102f976d79bceec1dd00b09eb03875be15cc24a548c89575bd115357ff860"}
Jul 07 15:14:30.571511 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:30.571426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t" event={"ID":"dbd0fb21-7b50-4317-bc2c-e9a903f633f1","Type":"ContainerStarted","Data":"755bcf3e4bb315f7eb47a1d5277d05e0d6bd9c6924f2aeaabfc0f035b7b946bb"}
Jul 07 15:14:30.571511 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:30.571482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t" event={"ID":"dbd0fb21-7b50-4317-bc2c-e9a903f633f1","Type":"ContainerStarted","Data":"509513497defa484303a4359befd6103694ca79b579317fef491c83b18d95ced"}
Jul 07 15:14:31.577628 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:31.577588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqx8f" event={"ID":"20b906c3-d257-41e0-bdeb-a28002532276","Type":"ContainerStarted","Data":"0bdabb1ff1b87c69a6170ebf4a2fcb6411d0c7d3cc7de8dd412b7e0663d84c9b"}
Jul 07 15:14:31.578092 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:31.577658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqx8f" event={"ID":"20b906c3-d257-41e0-bdeb-a28002532276","Type":"ContainerStarted","Data":"a21543ab785245a54c689f5910240b2f09d18f470955918d77c042e6430a6ef9"}
Jul 07 15:14:31.580415 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:31.580341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t" event={"ID":"dbd0fb21-7b50-4317-bc2c-e9a903f633f1","Type":"ContainerStarted","Data":"a870fc4d0e46237487a1e21770bba5ca47ca893877a115d870206f00d4b1e4fd"}
Jul 07 15:14:31.600179 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:31.600118    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nqx8f" podStartSLOduration=3.407720911 podStartE2EDuration="4.600101334s" totalImagesPullingTime="1.117500694s" totalInitContainerRuntime="74.879729ms" isStatefulPod=true podCreationTimestamp="2026-07-07 15:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:31.598534314 +0000 UTC m=+89.006643217" watchObservedRunningTime="2026-07-07 15:14:31.600101334 +0000 UTC m=+89.008210214"
Jul 07 15:14:31.619306 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:31.618113    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-hdf2t" podStartSLOduration=3.020974182 podStartE2EDuration="4.618098699s" totalImagesPullingTime="1.597124517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:31.61778657 +0000 UTC m=+89.025895475" watchObservedRunningTime="2026-07-07 15:14:31.618098699 +0000 UTC m=+89.026207602"
Jul 07 15:14:33.789332 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.789292    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:14:33.793828 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.793556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.797919 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.797491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 15:14:33.797919 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.797521    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-p2igct2gm79t\""
Jul 07 15:14:33.797919 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.797764    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 15:14:33.798959 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.798519    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-glfnz\""
Jul 07 15:14:33.798959 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.798760    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 15:14:33.799132 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.798972    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 15:14:33.799132 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.799026    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 15:14:33.799132 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.799078    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 15:14:33.799269 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.799166    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 15:14:33.799269 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.799230    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 15:14:33.799374 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.799231    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 15:14:33.799784 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.799606    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 15:14:33.799861 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.799799    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 15:14:33.800248 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.800113    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 15:14:33.800248 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.800140    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 15:14:33.803088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.802976    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 15:14:33.805280 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.805258    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 15:14:33.812198 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.812171    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:14:33.899335 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899665 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899665 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899665 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899665 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899619    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899665 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899653    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899775    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-config\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-config-out\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.899855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.900102 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-web-config\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.900102 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.900102 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.900102 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.899985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczrn\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-kube-api-access-mczrn\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:33.911726 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:33.911695    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:14:34.001288 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001288 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-config\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-config-out\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-web-config\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczrn\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-kube-api-access-mczrn\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.001587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.001818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.002043    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002856 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.002571    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.002982 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.002862    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.003564 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.003537    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.004306 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.004148    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.005507 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.005474    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.005782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.005755    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.006816 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.006791    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.007868 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.007842    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.007976 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.007848    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.007976 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.007895    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-config\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.007976 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.007924    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-config-out\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.009004 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.008982    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-web-config\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.009710 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.009668    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.011783 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.011743    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.012778 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.012694    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.012960 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.012877    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.014750 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.014704    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczrn\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-kube-api-access-mczrn\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.015073 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.015037    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.015997 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.015953    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:34.108886 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:34.108785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:14:38.925818 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:38.925767    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd" podUID="2192f76e-09d0-40c2-934a-02c8bcf14b12" containerName="registry" containerID="cri-o://5da8b92d1f32d3bde03b870ce4c61b1dcc4e96b802f272700dfc06fc042e0200" gracePeriod=30
Jul 07 15:14:41.625264 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:41.625224    2560 generic.go:356] "Generic (PLEG): container finished" podID="2192f76e-09d0-40c2-934a-02c8bcf14b12" containerID="5da8b92d1f32d3bde03b870ce4c61b1dcc4e96b802f272700dfc06fc042e0200" exitCode=0
Jul 07 15:14:41.625739 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:41.625282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd" event={"ID":"2192f76e-09d0-40c2-934a-02c8bcf14b12","Type":"ContainerDied","Data":"5da8b92d1f32d3bde03b870ce4c61b1dcc4e96b802f272700dfc06fc042e0200"}
Jul 07 15:14:42.003001 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.002974    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:14:42.175909 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.175879    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2192f76e-09d0-40c2-934a-02c8bcf14b12-ca-trust-extracted\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.175948    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.175991    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-trusted-ca\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.176016    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-certificates\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.176048    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-installation-pull-secrets\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176265 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.176081    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-image-registry-private-configuration\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176265 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.176128    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkjh4\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-kube-api-access-dkjh4\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176265 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.176183    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-bound-sa-token\") pod \"2192f76e-09d0-40c2-934a-02c8bcf14b12\" (UID: \"2192f76e-09d0-40c2-934a-02c8bcf14b12\") "
Jul 07 15:14:42.176970 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.176734    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:14:42.177044 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.177004    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:14:42.179534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.179492    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:14:42.179628 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.179546    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:14:42.180317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.180290    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-kube-api-access-dkjh4" (OuterVolumeSpecName: "kube-api-access-dkjh4") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "kube-api-access-dkjh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:14:42.180891 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.180848    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:14:42.181305 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.181279    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:14:42.186286 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.186262    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2192f76e-09d0-40c2-934a-02c8bcf14b12-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2192f76e-09d0-40c2-934a-02c8bcf14b12" (UID: "2192f76e-09d0-40c2-934a-02c8bcf14b12"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:14:42.277625 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277595    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjh4\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-kube-api-access-dkjh4\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.277751 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277630    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-bound-sa-token\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.277751 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277647    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2192f76e-09d0-40c2-934a-02c8bcf14b12-ca-trust-extracted\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.277751 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277685    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.277751 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277699    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-trusted-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.277751 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277711    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2192f76e-09d0-40c2-934a-02c8bcf14b12-registry-certificates\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.277751 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277723    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-installation-pull-secrets\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.277751 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.277737    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2192f76e-09d0-40c2-934a-02c8bcf14b12-image-registry-private-configuration\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:14:42.329843 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:14:42.329812    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a96036_c20e_45ff_95d9_d7e968d8ee67.slice/crio-f095e587ae8aba36886dc68a772fbc04cd9bfb5afca81fd7338fb8e99f0c21bd WatchSource:0}: Error finding container f095e587ae8aba36886dc68a772fbc04cd9bfb5afca81fd7338fb8e99f0c21bd: Status 404 returned error can't find the container with id f095e587ae8aba36886dc68a772fbc04cd9bfb5afca81fd7338fb8e99f0c21bd
Jul 07 15:14:42.331308 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.331288    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:14:42.630937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.630859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerStarted","Data":"f095e587ae8aba36886dc68a772fbc04cd9bfb5afca81fd7338fb8e99f0c21bd"}
Jul 07 15:14:42.632640 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.632611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9xbsw" event={"ID":"80bb84da-c2ad-47db-a15f-1df528b69cb0","Type":"ContainerStarted","Data":"7e0169eea48fdfd242d5e72a5b8bf73c11f062dc107877a372873a5a86ceb744"}
Jul 07 15:14:42.632869 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.632846    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-9xbsw"
Jul 07 15:14:42.633998 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.633967    2560 patch_prober.go:28] interesting pod/downloads-685bfb5f64-9xbsw container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused" start-of-body=
Jul 07 15:14:42.634111 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.634015    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-9xbsw" podUID="80bb84da-c2ad-47db-a15f-1df528b69cb0" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused"
Jul 07 15:14:42.634189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.634165    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"
Jul 07 15:14:42.634189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.634179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd" event={"ID":"2192f76e-09d0-40c2-934a-02c8bcf14b12","Type":"ContainerDied","Data":"e64ccadc62591bb63900c57d232ffa385f082b8ff36724186a6a094c5cbb840a"}
Jul 07 15:14:42.635973 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.635952    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556bd58ff-97kq7" event={"ID":"a6ac93f3-5480-48a3-8438-101ebf186d86","Type":"ContainerStarted","Data":"62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045"}
Jul 07 15:14:42.638929 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.638908    2560 scope.go:117] "RemoveContainer" containerID="5da8b92d1f32d3bde03b870ce4c61b1dcc4e96b802f272700dfc06fc042e0200"
Jul 07 15:14:42.671657 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.671603    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-9xbsw" podStartSLOduration=1.797913117 podStartE2EDuration="24.671582461s" totalImagesPullingTime="22.873669344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:42.671210753 +0000 UTC m=+100.079319655" watchObservedRunningTime="2026-07-07 15:14:42.671582461 +0000 UTC m=+100.079691363"
Jul 07 15:14:42.713988 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.713923    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556bd58ff-97kq7" podStartSLOduration=1.805677743 podStartE2EDuration="13.713904692s" totalImagesPullingTime="11.908226949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:42.712821872 +0000 UTC m=+100.120930776" watchObservedRunningTime="2026-07-07 15:14:42.713904692 +0000 UTC m=+100.122013595"
Jul 07 15:14:42.749366 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.749339    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"]
Jul 07 15:14:42.753882 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:42.753848    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7ff6ffdd6f-rf7rd"]
Jul 07 15:14:43.138371 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:43.138328    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2192f76e-09d0-40c2-934a-02c8bcf14b12" path="/var/lib/kubelet/pods/2192f76e-09d0-40c2-934a-02c8bcf14b12/volumes"
Jul 07 15:14:43.659473 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:43.659440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-9xbsw"
Jul 07 15:14:44.656274 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:44.656240    2560 generic.go:356] "Generic (PLEG): container finished" podID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerID="65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1" exitCode=0
Jul 07 15:14:44.656486 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:44.656334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1"}
Jul 07 15:14:46.483867 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:46.483837    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sfpgt"
Jul 07 15:14:49.681774 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:49.681732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerStarted","Data":"3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e"}
Jul 07 15:14:49.682189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:49.681800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerStarted","Data":"1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c"}
Jul 07 15:14:50.027623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:50.026771    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:50.027623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:50.026846    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:50.033737 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:50.033705    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:50.691427 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:50.691379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:14:51.172930 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:51.172895    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556bd58ff-97kq7"]
Jul 07 15:14:52.697841 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:52.697806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerStarted","Data":"ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4"}
Jul 07 15:14:52.698349 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:52.697902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerStarted","Data":"2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760"}
Jul 07 15:14:52.698349 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:52.697947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerStarted","Data":"482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac"}
Jul 07 15:14:52.698349 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:52.697976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerStarted","Data":"99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e"}
Jul 07 15:14:52.761290 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:52.761234    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=10.942833438 podStartE2EDuration="19.761217771s" totalImagesPullingTime="8.797459583s" totalInitContainerRuntime="20.92475ms" isStatefulPod=false podCreationTimestamp="2026-07-07 15:14:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:52.758678336 +0000 UTC m=+110.166787239" watchObservedRunningTime="2026-07-07 15:14:52.761217771 +0000 UTC m=+110.169326683"
Jul 07 15:14:54.109500 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:14:54.109461    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:15:17.717930 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:17.717881    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-556bd58ff-97kq7" podUID="a6ac93f3-5480-48a3-8438-101ebf186d86" containerName="console" containerID="cri-o://62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045" gracePeriod=15
Jul 07 15:15:17.993119 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:17.993043    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:15:18.078940 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.078897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-service-ca\") pod \"a6ac93f3-5480-48a3-8438-101ebf186d86\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") "
Jul 07 15:15:18.078940 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.078945    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-console-config\") pod \"a6ac93f3-5480-48a3-8438-101ebf186d86\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") "
Jul 07 15:15:18.079189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.078972    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-oauth-serving-cert\") pod \"a6ac93f3-5480-48a3-8438-101ebf186d86\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") "
Jul 07 15:15:18.079189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.079007    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-oauth-config\") pod \"a6ac93f3-5480-48a3-8438-101ebf186d86\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") "
Jul 07 15:15:18.079189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.079043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-serving-cert\") pod \"a6ac93f3-5480-48a3-8438-101ebf186d86\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") "
Jul 07 15:15:18.079189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.079075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7nwj\" (UniqueName: \"kubernetes.io/projected/a6ac93f3-5480-48a3-8438-101ebf186d86-kube-api-access-l7nwj\") pod \"a6ac93f3-5480-48a3-8438-101ebf186d86\" (UID: \"a6ac93f3-5480-48a3-8438-101ebf186d86\") "
Jul 07 15:15:18.079370 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.079325    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-console-config" (OuterVolumeSpecName: "console-config") pod "a6ac93f3-5480-48a3-8438-101ebf186d86" (UID: "a6ac93f3-5480-48a3-8438-101ebf186d86"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:15:18.079453 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.079371    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-service-ca" (OuterVolumeSpecName: "service-ca") pod "a6ac93f3-5480-48a3-8438-101ebf186d86" (UID: "a6ac93f3-5480-48a3-8438-101ebf186d86"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:15:18.079509 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.079458    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a6ac93f3-5480-48a3-8438-101ebf186d86" (UID: "a6ac93f3-5480-48a3-8438-101ebf186d86"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:15:18.081407 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.081362    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a6ac93f3-5480-48a3-8438-101ebf186d86" (UID: "a6ac93f3-5480-48a3-8438-101ebf186d86"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:15:18.081536 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.081474    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a6ac93f3-5480-48a3-8438-101ebf186d86" (UID: "a6ac93f3-5480-48a3-8438-101ebf186d86"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:15:18.081536 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.081495    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ac93f3-5480-48a3-8438-101ebf186d86-kube-api-access-l7nwj" (OuterVolumeSpecName: "kube-api-access-l7nwj") pod "a6ac93f3-5480-48a3-8438-101ebf186d86" (UID: "a6ac93f3-5480-48a3-8438-101ebf186d86"). InnerVolumeSpecName "kube-api-access-l7nwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:15:18.180520 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.180485    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:15:18.180520 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.180512    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-console-config\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:15:18.180520 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.180521    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac93f3-5480-48a3-8438-101ebf186d86-oauth-serving-cert\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:15:18.180520 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.180531    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-oauth-config\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:15:18.180779 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.180542    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ac93f3-5480-48a3-8438-101ebf186d86-console-serving-cert\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:15:18.180779 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.180552    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nwj\" (UniqueName: \"kubernetes.io/projected/a6ac93f3-5480-48a3-8438-101ebf186d86-kube-api-access-l7nwj\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:15:18.791573 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.791539    2560 generic.go:356] "Generic (PLEG): container finished" podID="a6ac93f3-5480-48a3-8438-101ebf186d86" containerID="62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045" exitCode=0
Jul 07 15:15:18.791967 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.791593    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556bd58ff-97kq7"
Jul 07 15:15:18.791967 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.791629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556bd58ff-97kq7" event={"ID":"a6ac93f3-5480-48a3-8438-101ebf186d86","Type":"ContainerDied","Data":"62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045"}
Jul 07 15:15:18.791967 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.791683    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556bd58ff-97kq7" event={"ID":"a6ac93f3-5480-48a3-8438-101ebf186d86","Type":"ContainerDied","Data":"f7a478bc95515985931bbe9a19249a2af604745f71b0eb5dbdaa72695a4eaafd"}
Jul 07 15:15:18.791967 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.791705    2560 scope.go:117] "RemoveContainer" containerID="62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045"
Jul 07 15:15:18.801310 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.801290    2560 scope.go:117] "RemoveContainer" containerID="62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045"
Jul 07 15:15:18.801678 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:15:18.801616    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045\": container with ID starting with 62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045 not found: ID does not exist" containerID="62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045"
Jul 07 15:15:18.801754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.801661    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045"} err="failed to get container status \"62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045\": rpc error: code = NotFound desc = could not find container \"62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045\": container with ID starting with 62ec6a76bb1da310c1a3f946b45d4e1e92cad538ffecfdeb4e1b2f208a5a2045 not found: ID does not exist"
Jul 07 15:15:18.822050 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.822027    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556bd58ff-97kq7"]
Jul 07 15:15:18.830661 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:18.830637    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556bd58ff-97kq7"]
Jul 07 15:15:19.136068 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:19.136042    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ac93f3-5480-48a3-8438-101ebf186d86" path="/var/lib/kubelet/pods/a6ac93f3-5480-48a3-8438-101ebf186d86/volumes"
Jul 07 15:15:34.109556 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:34.109517    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:15:34.189193 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:34.189159    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:15:34.869765 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:15:34.869741    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:10.883686 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:10.883651    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:16:10.884739 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:10.884685    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="prometheus" containerID="cri-o://1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c" gracePeriod=600
Jul 07 15:16:10.885155 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:10.885123    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4" gracePeriod=600
Jul 07 15:16:10.885248 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:10.885161    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-web" containerID="cri-o://482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac" gracePeriod=600
Jul 07 15:16:10.885301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:10.885135    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="thanos-sidecar" containerID="cri-o://99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e" gracePeriod=600
Jul 07 15:16:10.885301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:10.885281    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="config-reloader" containerID="cri-o://3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e" gracePeriod=600
Jul 07 15:16:10.886209 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:10.885508    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy" containerID="cri-o://2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760" gracePeriod=600
Jul 07 15:16:11.989952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.989921    2560 generic.go:356] "Generic (PLEG): container finished" podID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerID="ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4" exitCode=0
Jul 07 15:16:11.989952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.989945    2560 generic.go:356] "Generic (PLEG): container finished" podID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerID="2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760" exitCode=0
Jul 07 15:16:11.989952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.989953    2560 generic.go:356] "Generic (PLEG): container finished" podID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerID="99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e" exitCode=0
Jul 07 15:16:11.989952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.989960    2560 generic.go:356] "Generic (PLEG): container finished" podID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerID="3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e" exitCode=0
Jul 07 15:16:11.990571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.989965    2560 generic.go:356] "Generic (PLEG): container finished" podID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerID="1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c" exitCode=0
Jul 07 15:16:11.990571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.989966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4"}
Jul 07 15:16:11.990571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.990033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760"}
Jul 07 15:16:11.990571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.990060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e"}
Jul 07 15:16:11.990571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.990087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e"}
Jul 07 15:16:11.990571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:11.990120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c"}
Jul 07 15:16:12.136300 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.136276    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:12.213158 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213078    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-web-config\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213158 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213115    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-tls\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213158 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213144    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-2\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213184    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213371    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-serving-certs-ca-bundle\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213548 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213438    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-kubelet-serving-ca-bundle\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213548 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213548 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213498    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-grpc-tls\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213548 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213519    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.213548 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213530    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-trusted-ca-bundle\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213567    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-1\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213607    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-metrics-client-ca\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213646    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-config-out\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213677    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-thanos-prometheus-http-client-file\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-db\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213730    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-config\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.213782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213759    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mczrn\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-kube-api-access-mczrn\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.214090 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-metrics-client-certs\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.214090 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213832    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-kube-rbac-proxy\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.214090 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213862    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-tls-assets\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.214090 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213855    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.214090 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-0\") pod \"57a96036-c20e-45ff-95d9-d7e968d8ee67\" (UID: \"57a96036-c20e-45ff-95d9-d7e968d8ee67\") "
Jul 07 15:16:12.214090 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.213925    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.214376 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.214124    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.214376 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.214143    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.214376 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.214158    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.214548 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.214319    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.215895 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.215867    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.216234 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.216127    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.217490 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.217463    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:16:12.218689 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.218112    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.218689 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.218518    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.218689 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.218543    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.218903 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.218739    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.219061 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.219026    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.219137 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.219087    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.219786 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.219757    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.220939 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.220907    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-config-out" (OuterVolumeSpecName: "config-out") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:16:12.221632 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.221606    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-config" (OuterVolumeSpecName: "config") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.222005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.221796    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-kube-api-access-mczrn" (OuterVolumeSpecName: "kube-api-access-mczrn") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "kube-api-access-mczrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:16:12.222005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.221837    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:16:12.222292 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.222156    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.229920 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.229897    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-web-config" (OuterVolumeSpecName: "web-config") pod "57a96036-c20e-45ff-95d9-d7e968d8ee67" (UID: "57a96036-c20e-45ff-95d9-d7e968d8ee67"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:12.315025 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.314980    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315025 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315019    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-metrics-client-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315025 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315029    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-config-out\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315025 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315039    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315048    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-db\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315059    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-config\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315067    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczrn\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-kube-api-access-mczrn\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315076    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-metrics-client-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315084    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-kube-rbac-proxy\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315092    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57a96036-c20e-45ff-95d9-d7e968d8ee67-tls-assets\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315102    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315110    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-web-config\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315117    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315126    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315135    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57a96036-c20e-45ff-95d9-d7e968d8ee67-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315143    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.315297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.315152    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57a96036-c20e-45ff-95d9-d7e968d8ee67-secret-grpc-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:16:12.996170 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.996140    2560 generic.go:356] "Generic (PLEG): container finished" podID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerID="482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac" exitCode=0
Jul 07 15:16:12.996567 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.996182    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac"}
Jul 07 15:16:12.996567 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.996214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:12.996567 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.996239    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57a96036-c20e-45ff-95d9-d7e968d8ee67","Type":"ContainerDied","Data":"f095e587ae8aba36886dc68a772fbc04cd9bfb5afca81fd7338fb8e99f0c21bd"}
Jul 07 15:16:12.996567 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:12.996272    2560 scope.go:117] "RemoveContainer" containerID="ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4"
Jul 07 15:16:13.004432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.004413    2560 scope.go:117] "RemoveContainer" containerID="2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760"
Jul 07 15:16:13.011582 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.011409    2560 scope.go:117] "RemoveContainer" containerID="482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac"
Jul 07 15:16:13.017836 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.017821    2560 scope.go:117] "RemoveContainer" containerID="99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e"
Jul 07 15:16:13.024044 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.024029    2560 scope.go:117] "RemoveContainer" containerID="3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e"
Jul 07 15:16:13.030438 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.030421    2560 scope.go:117] "RemoveContainer" containerID="1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c"
Jul 07 15:16:13.030645 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.030627    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:16:13.037565 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.037547    2560 scope.go:117] "RemoveContainer" containerID="65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1"
Jul 07 15:16:13.044126 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.044110    2560 scope.go:117] "RemoveContainer" containerID="ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4"
Jul 07 15:16:13.044375 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:16:13.044357    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4\": container with ID starting with ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4 not found: ID does not exist" containerID="ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4"
Jul 07 15:16:13.044455 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.044382    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4"} err="failed to get container status \"ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4\": rpc error: code = NotFound desc = could not find container \"ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4\": container with ID starting with ecf3b56a3b731e4dbd93b2a14d0a2ab6e0821cee5e9b30c78b3c7f79096d89e4 not found: ID does not exist"
Jul 07 15:16:13.044455 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.044419    2560 scope.go:117] "RemoveContainer" containerID="2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760"
Jul 07 15:16:13.044714 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:16:13.044686    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760\": container with ID starting with 2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760 not found: ID does not exist" containerID="2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760"
Jul 07 15:16:13.044817 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.044721    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760"} err="failed to get container status \"2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760\": rpc error: code = NotFound desc = could not find container \"2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760\": container with ID starting with 2f478a07966026dc1c36e5d6e2387733bdb37ed819ce1480b8039ad59d5dd760 not found: ID does not exist"
Jul 07 15:16:13.044817 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.044746    2560 scope.go:117] "RemoveContainer" containerID="482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac"
Jul 07 15:16:13.045273 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:16:13.045245    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac\": container with ID starting with 482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac not found: ID does not exist" containerID="482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac"
Jul 07 15:16:13.045372 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.045302    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac"} err="failed to get container status \"482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac\": rpc error: code = NotFound desc = could not find container \"482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac\": container with ID starting with 482daed10795f2633fbeb58debfffb2f5643bb1f20951126adef5520fc1a8bac not found: ID does not exist"
Jul 07 15:16:13.045372 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.045323    2560 scope.go:117] "RemoveContainer" containerID="99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e"
Jul 07 15:16:13.045607 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:16:13.045590    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e\": container with ID starting with 99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e not found: ID does not exist" containerID="99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e"
Jul 07 15:16:13.045660 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.045615    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e"} err="failed to get container status \"99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e\": rpc error: code = NotFound desc = could not find container \"99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e\": container with ID starting with 99e92718844bd9a40ada412b7333b40f4619e4002b625ff409d201a179b8201e not found: ID does not exist"
Jul 07 15:16:13.045660 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.045630    2560 scope.go:117] "RemoveContainer" containerID="3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e"
Jul 07 15:16:13.045870 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:16:13.045854    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e\": container with ID starting with 3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e not found: ID does not exist" containerID="3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e"
Jul 07 15:16:13.045919 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.045875    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e"} err="failed to get container status \"3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e\": rpc error: code = NotFound desc = could not find container \"3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e\": container with ID starting with 3961c3876cdda963bf19e2421e5a3fb8dfe5d33564a23b84f2f9b0653db5930e not found: ID does not exist"
Jul 07 15:16:13.045919 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.045890    2560 scope.go:117] "RemoveContainer" containerID="1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c"
Jul 07 15:16:13.046117 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:16:13.046104    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c\": container with ID starting with 1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c not found: ID does not exist" containerID="1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c"
Jul 07 15:16:13.046161 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.046120    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c"} err="failed to get container status \"1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c\": rpc error: code = NotFound desc = could not find container \"1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c\": container with ID starting with 1b5e7d24e09933a538c51adf438cf3a25415f11afdedecef92abd9e6c420ad7c not found: ID does not exist"
Jul 07 15:16:13.046161 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.046132    2560 scope.go:117] "RemoveContainer" containerID="65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1"
Jul 07 15:16:13.046361 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:16:13.046343    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1\": container with ID starting with 65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1 not found: ID does not exist" containerID="65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1"
Jul 07 15:16:13.046438 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.046367    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1"} err="failed to get container status \"65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1\": rpc error: code = NotFound desc = could not find container \"65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1\": container with ID starting with 65ac98213cccde4fb214e968be74d815a03a80f94856b446cb385e9ae660d1e1 not found: ID does not exist"
Jul 07 15:16:13.046781 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.046763    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:16:13.084962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.084929    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:16:13.085171 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085159    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="thanos-sidecar"
Jul 07 15:16:13.085213 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085172    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="thanos-sidecar"
Jul 07 15:16:13.085213 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085182    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-thanos"
Jul 07 15:16:13.085213 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085187    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-thanos"
Jul 07 15:16:13.085213 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085195    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2192f76e-09d0-40c2-934a-02c8bcf14b12" containerName="registry"
Jul 07 15:16:13.085213 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085201    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2192f76e-09d0-40c2-934a-02c8bcf14b12" containerName="registry"
Jul 07 15:16:13.085213 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085208    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="config-reloader"
Jul 07 15:16:13.085213 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085213    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="config-reloader"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085225    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085230    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085237    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="init-config-reloader"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085241    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="init-config-reloader"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085248    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-web"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085254    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-web"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085289    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ac93f3-5480-48a3-8438-101ebf186d86" containerName="console"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085296    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ac93f3-5480-48a3-8438-101ebf186d86" containerName="console"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085302    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="prometheus"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085307    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="prometheus"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085368    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ac93f3-5480-48a3-8438-101ebf186d86" containerName="console"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085376    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2192f76e-09d0-40c2-934a-02c8bcf14b12" containerName="registry"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085382    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085400    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="thanos-sidecar"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085406    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="prometheus"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085411    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-web"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085415    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="kube-rbac-proxy-thanos"
Jul 07 15:16:13.085444 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.085445    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" containerName="config-reloader"
Jul 07 15:16:13.089334 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.089309    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.094996 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.094975    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 15:16:13.095440 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.095423    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-p2igct2gm79t\""
Jul 07 15:16:13.095716 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.095692    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-glfnz\""
Jul 07 15:16:13.095716 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.095712    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 15:16:13.095870 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.095692    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 15:16:13.096120 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.096104    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 15:16:13.096164 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.096154    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 15:16:13.096232 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.096217    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 15:16:13.096454 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.096439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 15:16:13.096506 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.096455    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 15:16:13.096857 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.096836    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 15:16:13.097092 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.097035    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 15:16:13.097092 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.097072    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 15:16:13.097320 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.097304    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 15:16:13.097423 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.097407    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 15:16:13.099433 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.099413    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 15:16:13.104826 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.104808    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 15:16:13.116654 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.116634    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:16:13.139094 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.136854    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57a96036-c20e-45ff-95d9-d7e968d8ee67" path="/var/lib/kubelet/pods/57a96036-c20e-45ff-95d9-d7e968d8ee67/volumes"
Jul 07 15:16:13.223648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223840    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-config\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223855    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.223884 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2h6\" (UniqueName: \"kubernetes.io/projected/b4955ebd-6d2c-4686-b712-055f09c0cbcf-kube-api-access-9z2h6\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4955ebd-6d2c-4686-b712-055f09c0cbcf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.223982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.224006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4955ebd-6d2c-4686-b712-055f09c0cbcf-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.224030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.224047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.224133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.224101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.324840 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.324800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.324840 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.324841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.324872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.324897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.324931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.324963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-config\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.324987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2h6\" (UniqueName: \"kubernetes.io/projected/b4955ebd-6d2c-4686-b712-055f09c0cbcf-kube-api-access-9z2h6\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4955ebd-6d2c-4686-b712-055f09c0cbcf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4955ebd-6d2c-4686-b712-055f09c0cbcf-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.325476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.326009 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.325683    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.326418 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.326373    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.328110 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.328082    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.328729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.328500    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4955ebd-6d2c-4686-b712-055f09c0cbcf-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.328729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.328541    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.328729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.328632    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-config\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.328933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.328748    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.328933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.328820    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.328933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.328855    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.329094 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.329032    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.329094 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.329078    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.329191 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.329109    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.329250 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.329232    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.329783 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.329760    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.330074 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.330048    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.330941 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.330920    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.331292 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.331273    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4955ebd-6d2c-4686-b712-055f09c0cbcf-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.331583 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.331567    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4955ebd-6d2c-4686-b712-055f09c0cbcf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.332483 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.332467    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4955ebd-6d2c-4686-b712-055f09c0cbcf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.339352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.339332    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2h6\" (UniqueName: \"kubernetes.io/projected/b4955ebd-6d2c-4686-b712-055f09c0cbcf-kube-api-access-9z2h6\") pod \"prometheus-k8s-0\" (UID: \"b4955ebd-6d2c-4686-b712-055f09c0cbcf\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.399566 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.399532    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:16:13.543449 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:13.543326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 15:16:13.545970 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:16:13.545946    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4955ebd_6d2c_4686_b712_055f09c0cbcf.slice/crio-6cda71221299b7a24e6fa44921a1e969228a099189b24b5a8d2f663d759263fe WatchSource:0}: Error finding container 6cda71221299b7a24e6fa44921a1e969228a099189b24b5a8d2f663d759263fe: Status 404 returned error can't find the container with id 6cda71221299b7a24e6fa44921a1e969228a099189b24b5a8d2f663d759263fe
Jul 07 15:16:14.003054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:14.003019    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4955ebd-6d2c-4686-b712-055f09c0cbcf" containerID="bee339d2935075fe362995bd39767ccee2395d9bb22c2b4eae4aae68bb56e0c9" exitCode=0
Jul 07 15:16:14.003475 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:14.003065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerDied","Data":"bee339d2935075fe362995bd39767ccee2395d9bb22c2b4eae4aae68bb56e0c9"}
Jul 07 15:16:14.003475 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:14.003116    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerStarted","Data":"6cda71221299b7a24e6fa44921a1e969228a099189b24b5a8d2f663d759263fe"}
Jul 07 15:16:15.010206 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:15.010175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerStarted","Data":"d880d2f077b135005b7758b2d692020374a235b2cfb3e8e2e12a7beb102e1344"}
Jul 07 15:16:15.010600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:15.010234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerStarted","Data":"50efbaf76f914d210e5a8e1fe6563052086c4aff568bbe731844a4d7105802b7"}
Jul 07 15:16:15.010600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:15.010260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerStarted","Data":"feb5c95c709a57ea3ae6925c1545faac65eb273a240e662ba87381b6d3301729"}
Jul 07 15:16:15.010600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:15.010284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerStarted","Data":"e29d81abc299b81bb53c7a974360fc1ee5dddcbace5b6804f4a49d9efd660310"}
Jul 07 15:16:15.010600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:15.010308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerStarted","Data":"1c071e350e0e4ce68605b7f97689d4287682f56117d4c5d48d506158915fa923"}
Jul 07 15:16:15.010600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:15.010334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4955ebd-6d2c-4686-b712-055f09c0cbcf","Type":"ContainerStarted","Data":"73204cc556f3d145a0bf24d237005bac8de88ff159048e0f0cc7eb42cf045736"}
Jul 07 15:16:15.047852 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:15.047792    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.029003768 podStartE2EDuration="2.047771346s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.767578ms" isStatefulPod=false podCreationTimestamp="2026-07-07 15:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:16:15.045573897 +0000 UTC m=+192.453682801" watchObservedRunningTime="2026-07-07 15:16:15.047771346 +0000 UTC m=+192.455880249"
Jul 07 15:16:18.400124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:16:18.400090    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:17:13.400556 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:17:13.400521    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:17:13.416011 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:17:13.415986    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:17:14.239190 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:17:14.239164    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 15:18:02.962244 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:02.962213    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 15:18:55.409209 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.409120    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qxxjf"]
Jul 07 15:18:55.411500 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.411478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.414208 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.414190    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 15:18:55.414337 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.414319    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-49gq5\""
Jul 07 15:18:55.415257 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.415242    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 15:18:55.424729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.424705    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qxxjf"]
Jul 07 15:18:55.475763 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.475728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4kk\" (UniqueName: \"kubernetes.io/projected/0bc68bed-dbe5-488d-894d-5f7df324a910-kube-api-access-nm4kk\") pod \"cert-manager-55b4d954d5-qxxjf\" (UID: \"0bc68bed-dbe5-488d-894d-5f7df324a910\") " pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.475763 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.475766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bc68bed-dbe5-488d-894d-5f7df324a910-bound-sa-token\") pod \"cert-manager-55b4d954d5-qxxjf\" (UID: \"0bc68bed-dbe5-488d-894d-5f7df324a910\") " pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.576797 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.576761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4kk\" (UniqueName: \"kubernetes.io/projected/0bc68bed-dbe5-488d-894d-5f7df324a910-kube-api-access-nm4kk\") pod \"cert-manager-55b4d954d5-qxxjf\" (UID: \"0bc68bed-dbe5-488d-894d-5f7df324a910\") " pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.576797 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.576803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bc68bed-dbe5-488d-894d-5f7df324a910-bound-sa-token\") pod \"cert-manager-55b4d954d5-qxxjf\" (UID: \"0bc68bed-dbe5-488d-894d-5f7df324a910\") " pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.586588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.586556    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bc68bed-dbe5-488d-894d-5f7df324a910-bound-sa-token\") pod \"cert-manager-55b4d954d5-qxxjf\" (UID: \"0bc68bed-dbe5-488d-894d-5f7df324a910\") " pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.586711 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.586687    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4kk\" (UniqueName: \"kubernetes.io/projected/0bc68bed-dbe5-488d-894d-5f7df324a910-kube-api-access-nm4kk\") pod \"cert-manager-55b4d954d5-qxxjf\" (UID: \"0bc68bed-dbe5-488d-894d-5f7df324a910\") " pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.720192 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.720103    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qxxjf"
Jul 07 15:18:55.851809 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.851756    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qxxjf"]
Jul 07 15:18:55.855277 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:18:55.855252    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bc68bed_dbe5_488d_894d_5f7df324a910.slice/crio-5b139c034a19737f177d1d788c4faf6d8da8f4429ea74477cd4123b04620d1ae WatchSource:0}: Error finding container 5b139c034a19737f177d1d788c4faf6d8da8f4429ea74477cd4123b04620d1ae: Status 404 returned error can't find the container with id 5b139c034a19737f177d1d788c4faf6d8da8f4429ea74477cd4123b04620d1ae
Jul 07 15:18:55.857074 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:55.857060    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:18:56.598404 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:56.598362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qxxjf" event={"ID":"0bc68bed-dbe5-488d-894d-5f7df324a910","Type":"ContainerStarted","Data":"5b139c034a19737f177d1d788c4faf6d8da8f4429ea74477cd4123b04620d1ae"}
Jul 07 15:18:59.612780 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:59.612741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qxxjf" event={"ID":"0bc68bed-dbe5-488d-894d-5f7df324a910","Type":"ContainerStarted","Data":"3cfa02c1dff0da77ba088e17c7542f6937c9434b88229a03e3ce4637254a471c"}
Jul 07 15:18:59.630437 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:18:59.630351    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qxxjf" podStartSLOduration=1.523601591 podStartE2EDuration="4.630331959s" totalImagesPullingTime="3.106730368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:18:59.630157799 +0000 UTC m=+357.038266701" watchObservedRunningTime="2026-07-07 15:18:59.630331959 +0000 UTC m=+357.038440862"
Jul 07 15:19:13.610342 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.610304    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"]
Jul 07 15:19:13.615526 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.615505    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.620290 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.620270    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 15:19:13.620433 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.620270    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rt5bv\""
Jul 07 15:19:13.620433 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.620270    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 15:19:13.621015 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.620995    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 15:19:13.621015 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.621010    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 15:19:13.621161 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.621144    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 15:19:13.630048 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.630029    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"]
Jul 07 15:19:13.714742 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.714710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88a46d28-1af0-48b4-a2a6-e32f72e0c237-metrics-cert\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.714928 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.714759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2x98\" (UniqueName: \"kubernetes.io/projected/88a46d28-1af0-48b4-a2a6-e32f72e0c237-kube-api-access-w2x98\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.714928 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.714815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88a46d28-1af0-48b4-a2a6-e32f72e0c237-cert\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.714928 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.714853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/88a46d28-1af0-48b4-a2a6-e32f72e0c237-manager-config\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.815860 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.815827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2x98\" (UniqueName: \"kubernetes.io/projected/88a46d28-1af0-48b4-a2a6-e32f72e0c237-kube-api-access-w2x98\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.815983 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.815866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88a46d28-1af0-48b4-a2a6-e32f72e0c237-cert\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.815983 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.815942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/88a46d28-1af0-48b4-a2a6-e32f72e0c237-manager-config\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.816064 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.816041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88a46d28-1af0-48b4-a2a6-e32f72e0c237-metrics-cert\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.816735 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.816711    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/88a46d28-1af0-48b4-a2a6-e32f72e0c237-manager-config\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.818328 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.818307    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88a46d28-1af0-48b4-a2a6-e32f72e0c237-metrics-cert\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.818424 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.818362    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88a46d28-1af0-48b4-a2a6-e32f72e0c237-cert\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.825194 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.825176    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2x98\" (UniqueName: \"kubernetes.io/projected/88a46d28-1af0-48b4-a2a6-e32f72e0c237-kube-api-access-w2x98\") pod \"lws-controller-manager-5ffd868d8b-jnf2r\" (UID: \"88a46d28-1af0-48b4-a2a6-e32f72e0c237\") " pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:13.923764 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:13.923734    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:14.074965 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:14.074927    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"]
Jul 07 15:19:14.078268 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:19:14.078234    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a46d28_1af0_48b4_a2a6_e32f72e0c237.slice/crio-529e91b2b96b0dd3463bb029b73b64fe78efa85dcf13649ca15cf3fab2ed3743 WatchSource:0}: Error finding container 529e91b2b96b0dd3463bb029b73b64fe78efa85dcf13649ca15cf3fab2ed3743: Status 404 returned error can't find the container with id 529e91b2b96b0dd3463bb029b73b64fe78efa85dcf13649ca15cf3fab2ed3743
Jul 07 15:19:14.665404 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:14.665368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r" event={"ID":"88a46d28-1af0-48b4-a2a6-e32f72e0c237","Type":"ContainerStarted","Data":"529e91b2b96b0dd3463bb029b73b64fe78efa85dcf13649ca15cf3fab2ed3743"}
Jul 07 15:19:19.686498 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:19.686463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r" event={"ID":"88a46d28-1af0-48b4-a2a6-e32f72e0c237","Type":"ContainerStarted","Data":"0edca4d36089991e5c1c8de74c91aeb560ae7bca79aace59a9593aac9ec18207"}
Jul 07 15:19:19.686862 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:19.686587    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:19.704580 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:19.704521    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r" podStartSLOduration=1.42840691 podStartE2EDuration="6.70450514s" totalImagesPullingTime="5.27609823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:19:19.703942086 +0000 UTC m=+377.112050989" watchObservedRunningTime="2026-07-07 15:19:19.70450514 +0000 UTC m=+377.112614074"
Jul 07 15:19:30.692524 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:30.692495    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5ffd868d8b-jnf2r"
Jul 07 15:19:37.253461 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.253424    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"]
Jul 07 15:19:37.256742 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.256726    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.261214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.261192    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 15:19:37.261724 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.261704    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 15:19:37.261846 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.261722    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 15:19:37.261846 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.261708    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 15:19:37.263248 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.263229    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rcsvm\""
Jul 07 15:19:37.273666 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.273644    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"]
Jul 07 15:19:37.415818 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.415784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ee9870b-660d-4abc-b940-0d35521a72e1-manifests\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.415992 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.415826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6z6t\" (UniqueName: \"kubernetes.io/projected/9ee9870b-660d-4abc-b940-0d35521a72e1-kube-api-access-n6z6t\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.415992 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.415905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ee9870b-660d-4abc-b940-0d35521a72e1-tmp\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.415992 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.415932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9870b-660d-4abc-b940-0d35521a72e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.415992 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.415965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ee9870b-660d-4abc-b940-0d35521a72e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.517066 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.516984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ee9870b-660d-4abc-b940-0d35521a72e1-manifests\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.517066 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.517026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z6t\" (UniqueName: \"kubernetes.io/projected/9ee9870b-660d-4abc-b940-0d35521a72e1-kube-api-access-n6z6t\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.517293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.517074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ee9870b-660d-4abc-b940-0d35521a72e1-tmp\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.517293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.517094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9870b-660d-4abc-b940-0d35521a72e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.517293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.517119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ee9870b-660d-4abc-b940-0d35521a72e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.517490 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.517414    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ee9870b-660d-4abc-b940-0d35521a72e1-manifests\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.519415 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.519373    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ee9870b-660d-4abc-b940-0d35521a72e1-tmp\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.519815 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.519792    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ee9870b-660d-4abc-b940-0d35521a72e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.519858 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.519792    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9870b-660d-4abc-b940-0d35521a72e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.542940 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.542912    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6z6t\" (UniqueName: \"kubernetes.io/projected/9ee9870b-660d-4abc-b940-0d35521a72e1-kube-api-access-n6z6t\") pod \"opendatahub-operator-controller-manager-dcbf69668-sf2w5\" (UID: \"9ee9870b-660d-4abc-b940-0d35521a72e1\") " pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.567427 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.567379    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:37.708111 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.708080    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"]
Jul 07 15:19:37.711570 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:19:37.711541    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee9870b_660d_4abc_b940_0d35521a72e1.slice/crio-09d4402bfb43ed87d73b60a218b9f1140ac6802d81f671ec399a17deee7ab16c WatchSource:0}: Error finding container 09d4402bfb43ed87d73b60a218b9f1140ac6802d81f671ec399a17deee7ab16c: Status 404 returned error can't find the container with id 09d4402bfb43ed87d73b60a218b9f1140ac6802d81f671ec399a17deee7ab16c
Jul 07 15:19:37.759432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:37.759379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5" event={"ID":"9ee9870b-660d-4abc-b940-0d35521a72e1","Type":"ContainerStarted","Data":"09d4402bfb43ed87d73b60a218b9f1140ac6802d81f671ec399a17deee7ab16c"}
Jul 07 15:19:41.777043 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:41.776955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5" event={"ID":"9ee9870b-660d-4abc-b940-0d35521a72e1","Type":"ContainerStarted","Data":"9905c86f63a9ed52fa5fb558456e8cd04ea6e624f7fb686a14b92881b99260b4"}
Jul 07 15:19:42.781477 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:42.781443    2560 generic.go:356] "Generic (PLEG): container finished" podID="9ee9870b-660d-4abc-b940-0d35521a72e1" containerID="9905c86f63a9ed52fa5fb558456e8cd04ea6e624f7fb686a14b92881b99260b4" exitCode=0
Jul 07 15:19:42.781880 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:42.781497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5" event={"ID":"9ee9870b-660d-4abc-b940-0d35521a72e1","Type":"ContainerDied","Data":"9905c86f63a9ed52fa5fb558456e8cd04ea6e624f7fb686a14b92881b99260b4"}
Jul 07 15:19:44.791355 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:44.791268    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5" event={"ID":"9ee9870b-660d-4abc-b940-0d35521a72e1","Type":"ContainerStarted","Data":"3b6915928b3fc70251ee4f50fc64f85f246f37da16a006c5125fc6b36ce55853"}
Jul 07 15:19:44.791355 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:44.791346    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:44.823972 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:44.823922    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5" podStartSLOduration=2.013984384 podStartE2EDuration="7.823905909s" totalImagesPullingTime="4.734940605s" totalInitContainerRuntime="1.07498092s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:19:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:19:44.823503279 +0000 UTC m=+402.231612181" watchObservedRunningTime="2026-07-07 15:19:44.823905909 +0000 UTC m=+402.232014811"
Jul 07 15:19:55.797228 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:55.797196    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dcbf69668-sf2w5"
Jul 07 15:19:58.750650 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.750618    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"]
Jul 07 15:19:58.752831 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.752816    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.755378 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.755348    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 15:19:58.755378 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.755359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 15:19:58.755617 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.755528    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mzmfk\""
Jul 07 15:19:58.756402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.756369    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 15:19:58.756402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.756378    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 15:19:58.778153 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.778117    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"]
Jul 07 15:19:58.784953 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.784928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77bp\" (UniqueName: \"kubernetes.io/projected/42f80fde-5172-4e59-a730-544ec86df5bc-kube-api-access-m77bp\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.785095 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.784967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f80fde-5172-4e59-a730-544ec86df5bc-tls-certs\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.785141 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.785092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f80fde-5172-4e59-a730-544ec86df5bc-tmp\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.885708 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.885672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f80fde-5172-4e59-a730-544ec86df5bc-tls-certs\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.885896 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.885735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f80fde-5172-4e59-a730-544ec86df5bc-tmp\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.885896 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.885778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77bp\" (UniqueName: \"kubernetes.io/projected/42f80fde-5172-4e59-a730-544ec86df5bc-kube-api-access-m77bp\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.885896 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:19:58.885815    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 07 15:19:58.885896 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:19:58.885872    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f80fde-5172-4e59-a730-544ec86df5bc-tls-certs podName:42f80fde-5172-4e59-a730-544ec86df5bc nodeName:}" failed. No retries permitted until 2026-07-07 15:19:59.385852977 +0000 UTC m=+416.793961857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42f80fde-5172-4e59-a730-544ec86df5bc-tls-certs") pod "kube-auth-proxy-5c6c759bc-8f6cn" (UID: "42f80fde-5172-4e59-a730-544ec86df5bc") : secret "kube-auth-proxy-tls" not found
Jul 07 15:19:58.888184 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.888156    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f80fde-5172-4e59-a730-544ec86df5bc-tmp\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:58.901423 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:58.901377    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77bp\" (UniqueName: \"kubernetes.io/projected/42f80fde-5172-4e59-a730-544ec86df5bc-kube-api-access-m77bp\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:59.389873 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:59.389837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f80fde-5172-4e59-a730-544ec86df5bc-tls-certs\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:59.392467 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:59.392443    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f80fde-5172-4e59-a730-544ec86df5bc-tls-certs\") pod \"kube-auth-proxy-5c6c759bc-8f6cn\" (UID: \"42f80fde-5172-4e59-a730-544ec86df5bc\") " pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:59.663070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:59.662991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"
Jul 07 15:19:59.807569 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:59.807542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn"]
Jul 07 15:19:59.809725 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:19:59.809697    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f80fde_5172_4e59_a730_544ec86df5bc.slice/crio-2372d4258e82d35d81f46de0cae683703a4678bb540080bc23c25a04096b53c7 WatchSource:0}: Error finding container 2372d4258e82d35d81f46de0cae683703a4678bb540080bc23c25a04096b53c7: Status 404 returned error can't find the container with id 2372d4258e82d35d81f46de0cae683703a4678bb540080bc23c25a04096b53c7
Jul 07 15:19:59.849347 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:19:59.849317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn" event={"ID":"42f80fde-5172-4e59-a730-544ec86df5bc","Type":"ContainerStarted","Data":"2372d4258e82d35d81f46de0cae683703a4678bb540080bc23c25a04096b53c7"}
Jul 07 15:20:03.870185 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:20:03.870147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn" event={"ID":"42f80fde-5172-4e59-a730-544ec86df5bc","Type":"ContainerStarted","Data":"5bc18cdb3cb54c9ca5003d336e519fe7842b785c3001c5d567237be76129b87e"}
Jul 07 15:20:03.901209 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:20:03.901151    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c6c759bc-8f6cn" podStartSLOduration=2.800959378 podStartE2EDuration="5.901132806s" totalImagesPullingTime="3.100173428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:20:03.899899391 +0000 UTC m=+421.308008293" watchObservedRunningTime="2026-07-07 15:20:03.901132806 +0000 UTC m=+421.309241709"
Jul 07 15:22:30.636237 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.636203    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"]
Jul 07 15:22:30.642654 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.642628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:30.645204 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.645177    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 15:22:30.645330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.645265    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 15:22:30.645402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.645368    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 15:22:30.646317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.646298    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z8hxh\""
Jul 07 15:22:30.654402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.654363    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"]
Jul 07 15:22:30.739675 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.739642    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"]
Jul 07 15:22:30.773237 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.773204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k7kd\" (UniqueName: \"kubernetes.io/projected/06177451-863c-40be-93fc-62358dab5599-kube-api-access-6k7kd\") pod \"limitador-limitador-64c94fd64f-f5pqd\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:30.773419 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.773272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06177451-863c-40be-93fc-62358dab5599-config-file\") pod \"limitador-limitador-64c94fd64f-f5pqd\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:30.874383 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.874343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06177451-863c-40be-93fc-62358dab5599-config-file\") pod \"limitador-limitador-64c94fd64f-f5pqd\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:30.874617 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.874439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7kd\" (UniqueName: \"kubernetes.io/projected/06177451-863c-40be-93fc-62358dab5599-kube-api-access-6k7kd\") pod \"limitador-limitador-64c94fd64f-f5pqd\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:30.875070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.875044    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06177451-863c-40be-93fc-62358dab5599-config-file\") pod \"limitador-limitador-64c94fd64f-f5pqd\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:30.884183 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.884149    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7kd\" (UniqueName: \"kubernetes.io/projected/06177451-863c-40be-93fc-62358dab5599-kube-api-access-6k7kd\") pod \"limitador-limitador-64c94fd64f-f5pqd\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:30.953461 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:30.953337    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:31.087204 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.087155    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"]
Jul 07 15:22:31.090031 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:22:31.090002    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06177451_863c_40be_93fc_62358dab5599.slice/crio-417ed4c474f88337e3f55ea26ebb303a08bb851607147a6b4dfad3adf3aea78f WatchSource:0}: Error finding container 417ed4c474f88337e3f55ea26ebb303a08bb851607147a6b4dfad3adf3aea78f: Status 404 returned error can't find the container with id 417ed4c474f88337e3f55ea26ebb303a08bb851607147a6b4dfad3adf3aea78f
Jul 07 15:22:31.455753 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.455710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd" event={"ID":"06177451-863c-40be-93fc-62358dab5599","Type":"ContainerStarted","Data":"417ed4c474f88337e3f55ea26ebb303a08bb851607147a6b4dfad3adf3aea78f"}
Jul 07 15:22:31.527798 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.527765    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wg66v"]
Jul 07 15:22:31.530451 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.530431    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wg66v"
Jul 07 15:22:31.533701 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.533681    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h86w4\""
Jul 07 15:22:31.550770 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.550748    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wg66v"]
Jul 07 15:22:31.682374 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.682339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjf84\" (UniqueName: \"kubernetes.io/projected/ec2c1e54-c500-4b64-92c2-db3e2f7dadb9-kube-api-access-jjf84\") pod \"authorino-58cd955675-wg66v\" (UID: \"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9\") " pod="kuadrant-system/authorino-58cd955675-wg66v"
Jul 07 15:22:31.783277 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.783198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjf84\" (UniqueName: \"kubernetes.io/projected/ec2c1e54-c500-4b64-92c2-db3e2f7dadb9-kube-api-access-jjf84\") pod \"authorino-58cd955675-wg66v\" (UID: \"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9\") " pod="kuadrant-system/authorino-58cd955675-wg66v"
Jul 07 15:22:31.792330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.792297    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjf84\" (UniqueName: \"kubernetes.io/projected/ec2c1e54-c500-4b64-92c2-db3e2f7dadb9-kube-api-access-jjf84\") pod \"authorino-58cd955675-wg66v\" (UID: \"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9\") " pod="kuadrant-system/authorino-58cd955675-wg66v"
Jul 07 15:22:31.839776 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.839744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wg66v"
Jul 07 15:22:31.967595 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.966113    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-ch8m5"]
Jul 07 15:22:31.969098 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.969060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ch8m5"
Jul 07 15:22:31.984780 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:31.984747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ch8m5"]
Jul 07 15:22:32.025798 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.025756    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wg66v"]
Jul 07 15:22:32.035859 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:22:32.035786    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec2c1e54_c500_4b64_92c2_db3e2f7dadb9.slice/crio-811468ec32541ed7f3783d8c9b3697f75ca94e4d539dc8f31b6505120593b4b1 WatchSource:0}: Error finding container 811468ec32541ed7f3783d8c9b3697f75ca94e4d539dc8f31b6505120593b4b1: Status 404 returned error can't find the container with id 811468ec32541ed7f3783d8c9b3697f75ca94e4d539dc8f31b6505120593b4b1
Jul 07 15:22:32.085944 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.085911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncw5s\" (UniqueName: \"kubernetes.io/projected/e0303433-4610-4ee5-bedf-d35c7c2e110d-kube-api-access-ncw5s\") pod \"authorino-75d556bd77-ch8m5\" (UID: \"e0303433-4610-4ee5-bedf-d35c7c2e110d\") " pod="kuadrant-system/authorino-75d556bd77-ch8m5"
Jul 07 15:22:32.186460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.186419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncw5s\" (UniqueName: \"kubernetes.io/projected/e0303433-4610-4ee5-bedf-d35c7c2e110d-kube-api-access-ncw5s\") pod \"authorino-75d556bd77-ch8m5\" (UID: \"e0303433-4610-4ee5-bedf-d35c7c2e110d\") " pod="kuadrant-system/authorino-75d556bd77-ch8m5"
Jul 07 15:22:32.196323 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.196269    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncw5s\" (UniqueName: \"kubernetes.io/projected/e0303433-4610-4ee5-bedf-d35c7c2e110d-kube-api-access-ncw5s\") pod \"authorino-75d556bd77-ch8m5\" (UID: \"e0303433-4610-4ee5-bedf-d35c7c2e110d\") " pod="kuadrant-system/authorino-75d556bd77-ch8m5"
Jul 07 15:22:32.287908 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.287825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ch8m5"
Jul 07 15:22:32.443791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.443758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ch8m5"]
Jul 07 15:22:32.447133 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:22:32.447100    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0303433_4610_4ee5_bedf_d35c7c2e110d.slice/crio-fd48ce7659e4d08ddd8523607c2a790f2260b93e47bdbb452830960bf2b0d41d WatchSource:0}: Error finding container fd48ce7659e4d08ddd8523607c2a790f2260b93e47bdbb452830960bf2b0d41d: Status 404 returned error can't find the container with id fd48ce7659e4d08ddd8523607c2a790f2260b93e47bdbb452830960bf2b0d41d
Jul 07 15:22:32.461797 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.461766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ch8m5" event={"ID":"e0303433-4610-4ee5-bedf-d35c7c2e110d","Type":"ContainerStarted","Data":"fd48ce7659e4d08ddd8523607c2a790f2260b93e47bdbb452830960bf2b0d41d"}
Jul 07 15:22:32.463155 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:32.463129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wg66v" event={"ID":"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9","Type":"ContainerStarted","Data":"811468ec32541ed7f3783d8c9b3697f75ca94e4d539dc8f31b6505120593b4b1"}
Jul 07 15:22:36.483084 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.483051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ch8m5" event={"ID":"e0303433-4610-4ee5-bedf-d35c7c2e110d","Type":"ContainerStarted","Data":"eec2c91f327cc450c57b345fa94da8ff9f176d6f9aedb0f1ba9ca4ad43512431"}
Jul 07 15:22:36.484613 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.484591    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd" event={"ID":"06177451-863c-40be-93fc-62358dab5599","Type":"ContainerStarted","Data":"0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563"}
Jul 07 15:22:36.484719 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.484635    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:36.485980 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.485955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wg66v" event={"ID":"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9","Type":"ContainerStarted","Data":"40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2"}
Jul 07 15:22:36.499657 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.499613    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-ch8m5" podStartSLOduration=2.149291554 podStartE2EDuration="5.499600651s" totalImagesPullingTime="3.350309097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:22:36.498091977 +0000 UTC m=+573.906200881" watchObservedRunningTime="2026-07-07 15:22:36.499600651 +0000 UTC m=+573.907709605"
Jul 07 15:22:36.517346 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.517296    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd" podStartSLOduration=1.85103263 podStartE2EDuration="6.517279863s" totalImagesPullingTime="4.666247233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:22:36.516149388 +0000 UTC m=+573.924258289" watchObservedRunningTime="2026-07-07 15:22:36.517279863 +0000 UTC m=+573.925388765"
Jul 07 15:22:36.529048 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.529015    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wg66v"]
Jul 07 15:22:36.540711 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:36.540661    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wg66v" podStartSLOduration=1.7849996830000001 podStartE2EDuration="5.540642903s" totalImagesPullingTime="3.75564322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:22:36.539684872 +0000 UTC m=+573.947793775" watchObservedRunningTime="2026-07-07 15:22:36.540642903 +0000 UTC m=+573.948751805"
Jul 07 15:22:38.493535 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:38.493492    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wg66v" podUID="ec2c1e54-c500-4b64-92c2-db3e2f7dadb9" containerName="authorino" containerID="cri-o://40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2" gracePeriod=30
Jul 07 15:22:38.741292 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:38.741269    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wg66v"
Jul 07 15:22:38.849070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:38.848989    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjf84\" (UniqueName: \"kubernetes.io/projected/ec2c1e54-c500-4b64-92c2-db3e2f7dadb9-kube-api-access-jjf84\") pod \"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9\" (UID: \"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9\") "
Jul 07 15:22:38.851153 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:38.851117    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2c1e54-c500-4b64-92c2-db3e2f7dadb9-kube-api-access-jjf84" (OuterVolumeSpecName: "kube-api-access-jjf84") pod "ec2c1e54-c500-4b64-92c2-db3e2f7dadb9" (UID: "ec2c1e54-c500-4b64-92c2-db3e2f7dadb9"). InnerVolumeSpecName "kube-api-access-jjf84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:22:38.949893 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:38.949858    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjf84\" (UniqueName: \"kubernetes.io/projected/ec2c1e54-c500-4b64-92c2-db3e2f7dadb9-kube-api-access-jjf84\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:22:39.498652 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.498569    2560 generic.go:356] "Generic (PLEG): container finished" podID="ec2c1e54-c500-4b64-92c2-db3e2f7dadb9" containerID="40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2" exitCode=0
Jul 07 15:22:39.498652 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.498616    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wg66v"
Jul 07 15:22:39.498652 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.498642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wg66v" event={"ID":"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9","Type":"ContainerDied","Data":"40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2"}
Jul 07 15:22:39.499114 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.498674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wg66v" event={"ID":"ec2c1e54-c500-4b64-92c2-db3e2f7dadb9","Type":"ContainerDied","Data":"811468ec32541ed7f3783d8c9b3697f75ca94e4d539dc8f31b6505120593b4b1"}
Jul 07 15:22:39.499114 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.498690    2560 scope.go:117] "RemoveContainer" containerID="40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2"
Jul 07 15:22:39.506669 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.506650    2560 scope.go:117] "RemoveContainer" containerID="40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2"
Jul 07 15:22:39.506960 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:22:39.506934    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2\": container with ID starting with 40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2 not found: ID does not exist" containerID="40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2"
Jul 07 15:22:39.507013 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.506962    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2"} err="failed to get container status \"40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2\": rpc error: code = NotFound desc = could not find container \"40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2\": container with ID starting with 40d078934a4b1f956c0b662bbb707965a41d85833dff997f954f35e2971173f2 not found: ID does not exist"
Jul 07 15:22:39.516534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.516509    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wg66v"]
Jul 07 15:22:39.520233 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:39.520212    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wg66v"]
Jul 07 15:22:41.136845 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:41.136809    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2c1e54-c500-4b64-92c2-db3e2f7dadb9" path="/var/lib/kubelet/pods/ec2c1e54-c500-4b64-92c2-db3e2f7dadb9/volumes"
Jul 07 15:22:45.721189 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:45.721156    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"]
Jul 07 15:22:45.721685 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:45.721349    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd" podUID="06177451-863c-40be-93fc-62358dab5599" containerName="limitador" containerID="cri-o://0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563" gracePeriod=30
Jul 07 15:22:45.722067 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:45.721960    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:46.263319 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.263293    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:46.296284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.296199    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k7kd\" (UniqueName: \"kubernetes.io/projected/06177451-863c-40be-93fc-62358dab5599-kube-api-access-6k7kd\") pod \"06177451-863c-40be-93fc-62358dab5599\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") "
Jul 07 15:22:46.296467 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.296302    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06177451-863c-40be-93fc-62358dab5599-config-file\") pod \"06177451-863c-40be-93fc-62358dab5599\" (UID: \"06177451-863c-40be-93fc-62358dab5599\") "
Jul 07 15:22:46.296657 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.296633    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06177451-863c-40be-93fc-62358dab5599-config-file" (OuterVolumeSpecName: "config-file") pod "06177451-863c-40be-93fc-62358dab5599" (UID: "06177451-863c-40be-93fc-62358dab5599"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:22:46.298326 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.298300    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06177451-863c-40be-93fc-62358dab5599-kube-api-access-6k7kd" (OuterVolumeSpecName: "kube-api-access-6k7kd") pod "06177451-863c-40be-93fc-62358dab5599" (UID: "06177451-863c-40be-93fc-62358dab5599"). InnerVolumeSpecName "kube-api-access-6k7kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:22:46.397199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.397164    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k7kd\" (UniqueName: \"kubernetes.io/projected/06177451-863c-40be-93fc-62358dab5599-kube-api-access-6k7kd\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:22:46.397199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.397193    2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06177451-863c-40be-93fc-62358dab5599-config-file\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:22:46.529042 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.529007    2560 generic.go:356] "Generic (PLEG): container finished" podID="06177451-863c-40be-93fc-62358dab5599" containerID="0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563" exitCode=0
Jul 07 15:22:46.529217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.529059    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"
Jul 07 15:22:46.529217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.529083    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd" event={"ID":"06177451-863c-40be-93fc-62358dab5599","Type":"ContainerDied","Data":"0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563"}
Jul 07 15:22:46.529217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.529129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5pqd" event={"ID":"06177451-863c-40be-93fc-62358dab5599","Type":"ContainerDied","Data":"417ed4c474f88337e3f55ea26ebb303a08bb851607147a6b4dfad3adf3aea78f"}
Jul 07 15:22:46.529217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.529153    2560 scope.go:117] "RemoveContainer" containerID="0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563"
Jul 07 15:22:46.538449 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.538433    2560 scope.go:117] "RemoveContainer" containerID="0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563"
Jul 07 15:22:46.538725 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:22:46.538707    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563\": container with ID starting with 0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563 not found: ID does not exist" containerID="0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563"
Jul 07 15:22:46.538778 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.538736    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563"} err="failed to get container status \"0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563\": rpc error: code = NotFound desc = could not find container \"0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563\": container with ID starting with 0339c4b4739443837f863ae56def8e530a6786d9fdc4bcc1037352ca689ee563 not found: ID does not exist"
Jul 07 15:22:46.554423 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.554336    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"]
Jul 07 15:22:46.558472 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:46.558451    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5pqd"]
Jul 07 15:22:47.137719 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.137689    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06177451-863c-40be-93fc-62358dab5599" path="/var/lib/kubelet/pods/06177451-863c-40be-93fc-62358dab5599/volumes"
Jul 07 15:22:47.274978 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.274947    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-h26ct"]
Jul 07 15:22:47.275244 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.275233    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06177451-863c-40be-93fc-62358dab5599" containerName="limitador"
Jul 07 15:22:47.275296 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.275245    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06177451-863c-40be-93fc-62358dab5599" containerName="limitador"
Jul 07 15:22:47.275296 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.275254    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2c1e54-c500-4b64-92c2-db3e2f7dadb9" containerName="authorino"
Jul 07 15:22:47.275296 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.275260    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2c1e54-c500-4b64-92c2-db3e2f7dadb9" containerName="authorino"
Jul 07 15:22:47.275402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.275317    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06177451-863c-40be-93fc-62358dab5599" containerName="limitador"
Jul 07 15:22:47.275402 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.275325    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2c1e54-c500-4b64-92c2-db3e2f7dadb9" containerName="authorino"
Jul 07 15:22:47.278779 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.278761    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.281628 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.281605    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 07 15:22:47.282109 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.282092    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4dpgr\""
Jul 07 15:22:47.291056 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.291032    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-h26ct"]
Jul 07 15:22:47.305070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.305045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78a797b5-513d-471b-acaf-7950a41d5b71-data\") pod \"postgres-69fb4594cf-h26ct\" (UID: \"78a797b5-513d-471b-acaf-7950a41d5b71\") " pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.305194 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.305086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6dr5\" (UniqueName: \"kubernetes.io/projected/78a797b5-513d-471b-acaf-7950a41d5b71-kube-api-access-g6dr5\") pod \"postgres-69fb4594cf-h26ct\" (UID: \"78a797b5-513d-471b-acaf-7950a41d5b71\") " pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.406317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.406235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78a797b5-513d-471b-acaf-7950a41d5b71-data\") pod \"postgres-69fb4594cf-h26ct\" (UID: \"78a797b5-513d-471b-acaf-7950a41d5b71\") " pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.406317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.406279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6dr5\" (UniqueName: \"kubernetes.io/projected/78a797b5-513d-471b-acaf-7950a41d5b71-kube-api-access-g6dr5\") pod \"postgres-69fb4594cf-h26ct\" (UID: \"78a797b5-513d-471b-acaf-7950a41d5b71\") " pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.406711 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.406688    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78a797b5-513d-471b-acaf-7950a41d5b71-data\") pod \"postgres-69fb4594cf-h26ct\" (UID: \"78a797b5-513d-471b-acaf-7950a41d5b71\") " pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.422082 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.422050    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6dr5\" (UniqueName: \"kubernetes.io/projected/78a797b5-513d-471b-acaf-7950a41d5b71-kube-api-access-g6dr5\") pod \"postgres-69fb4594cf-h26ct\" (UID: \"78a797b5-513d-471b-acaf-7950a41d5b71\") " pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.588772 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.588735    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:47.729301 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:47.729127    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-h26ct"]
Jul 07 15:22:47.732083 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:22:47.732048    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a797b5_513d_471b_acaf_7950a41d5b71.slice/crio-16a4213d08a614bc07b462e15ad9a110c71f688cfe6ffa638f859881bb3441bb WatchSource:0}: Error finding container 16a4213d08a614bc07b462e15ad9a110c71f688cfe6ffa638f859881bb3441bb: Status 404 returned error can't find the container with id 16a4213d08a614bc07b462e15ad9a110c71f688cfe6ffa638f859881bb3441bb
Jul 07 15:22:48.540231 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:48.540196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-h26ct" event={"ID":"78a797b5-513d-471b-acaf-7950a41d5b71","Type":"ContainerStarted","Data":"16a4213d08a614bc07b462e15ad9a110c71f688cfe6ffa638f859881bb3441bb"}
Jul 07 15:22:53.563855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:53.563821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-h26ct" event={"ID":"78a797b5-513d-471b-acaf-7950a41d5b71","Type":"ContainerStarted","Data":"88555608eb88e01a4691e65d27e9d73ade348823d9a5515aac5d4065b8b2f141"}
Jul 07 15:22:53.564241 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:53.563921    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:22:53.584732 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:53.584680    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-h26ct" podStartSLOduration=1.5476393769999999 podStartE2EDuration="6.584666125s" totalImagesPullingTime="5.037026748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:22:53.58301444 +0000 UTC m=+590.991123355" watchObservedRunningTime="2026-07-07 15:22:53.584666125 +0000 UTC m=+590.992775027"
Jul 07 15:22:59.598489 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:22:59.598456    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-h26ct"
Jul 07 15:23:00.198291 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.198251    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-65vtm"]
Jul 07 15:23:00.205188 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.205161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-65vtm"
Jul 07 15:23:00.221185 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.221158    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-65vtm"]
Jul 07 15:23:00.324695 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.324654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8l4x\" (UniqueName: \"kubernetes.io/projected/b59307d3-ad78-4a5e-a43f-e8176f9629af-kube-api-access-w8l4x\") pod \"authorino-5d4b67c446-65vtm\" (UID: \"b59307d3-ad78-4a5e-a43f-e8176f9629af\") " pod="kuadrant-system/authorino-5d4b67c446-65vtm"
Jul 07 15:23:00.426130 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.426092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8l4x\" (UniqueName: \"kubernetes.io/projected/b59307d3-ad78-4a5e-a43f-e8176f9629af-kube-api-access-w8l4x\") pod \"authorino-5d4b67c446-65vtm\" (UID: \"b59307d3-ad78-4a5e-a43f-e8176f9629af\") " pod="kuadrant-system/authorino-5d4b67c446-65vtm"
Jul 07 15:23:00.441317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.441291    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8l4x\" (UniqueName: \"kubernetes.io/projected/b59307d3-ad78-4a5e-a43f-e8176f9629af-kube-api-access-w8l4x\") pod \"authorino-5d4b67c446-65vtm\" (UID: \"b59307d3-ad78-4a5e-a43f-e8176f9629af\") " pod="kuadrant-system/authorino-5d4b67c446-65vtm"
Jul 07 15:23:00.468786 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.468706    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-65vtm"]
Jul 07 15:23:00.468963 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.468952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-65vtm"
Jul 07 15:23:00.604855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:00.604830    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-65vtm"]
Jul 07 15:23:00.607272 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:23:00.607244    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb59307d3_ad78_4a5e_a43f_e8176f9629af.slice/crio-b510de63d5e0a384820b6f5947eeffe4ea3021153dc0d3d74552d64a7be48013 WatchSource:0}: Error finding container b510de63d5e0a384820b6f5947eeffe4ea3021153dc0d3d74552d64a7be48013: Status 404 returned error can't find the container with id b510de63d5e0a384820b6f5947eeffe4ea3021153dc0d3d74552d64a7be48013
Jul 07 15:23:01.599983 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:01.599945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-65vtm" event={"ID":"b59307d3-ad78-4a5e-a43f-e8176f9629af","Type":"ContainerStarted","Data":"b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6"}
Jul 07 15:23:01.599983 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:01.599974    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-65vtm" podUID="b59307d3-ad78-4a5e-a43f-e8176f9629af" containerName="authorino" containerID="cri-o://b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6" gracePeriod=30
Jul 07 15:23:01.600303 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:01.599994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-65vtm" event={"ID":"b59307d3-ad78-4a5e-a43f-e8176f9629af","Type":"ContainerStarted","Data":"b510de63d5e0a384820b6f5947eeffe4ea3021153dc0d3d74552d64a7be48013"}
Jul 07 15:23:01.616318 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:01.616272    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-65vtm" podStartSLOduration=1.166754409 podStartE2EDuration="1.616254997s" totalImagesPullingTime="449.500588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:01.615448502 +0000 UTC m=+599.023557405" watchObservedRunningTime="2026-07-07 15:23:01.616254997 +0000 UTC m=+599.024363932"
Jul 07 15:23:01.847202 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:01.847156    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-65vtm"
Jul 07 15:23:01.938608 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:01.938576    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8l4x\" (UniqueName: \"kubernetes.io/projected/b59307d3-ad78-4a5e-a43f-e8176f9629af-kube-api-access-w8l4x\") pod \"b59307d3-ad78-4a5e-a43f-e8176f9629af\" (UID: \"b59307d3-ad78-4a5e-a43f-e8176f9629af\") "
Jul 07 15:23:01.940609 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:01.940586    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59307d3-ad78-4a5e-a43f-e8176f9629af-kube-api-access-w8l4x" (OuterVolumeSpecName: "kube-api-access-w8l4x") pod "b59307d3-ad78-4a5e-a43f-e8176f9629af" (UID: "b59307d3-ad78-4a5e-a43f-e8176f9629af"). InnerVolumeSpecName "kube-api-access-w8l4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:23:02.039128 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.039088    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8l4x\" (UniqueName: \"kubernetes.io/projected/b59307d3-ad78-4a5e-a43f-e8176f9629af-kube-api-access-w8l4x\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:23:02.605120 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.605083    2560 generic.go:356] "Generic (PLEG): container finished" podID="b59307d3-ad78-4a5e-a43f-e8176f9629af" containerID="b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6" exitCode=0
Jul 07 15:23:02.605293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.605125    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-65vtm"
Jul 07 15:23:02.605293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.605172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-65vtm" event={"ID":"b59307d3-ad78-4a5e-a43f-e8176f9629af","Type":"ContainerDied","Data":"b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6"}
Jul 07 15:23:02.605293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.605228    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-65vtm" event={"ID":"b59307d3-ad78-4a5e-a43f-e8176f9629af","Type":"ContainerDied","Data":"b510de63d5e0a384820b6f5947eeffe4ea3021153dc0d3d74552d64a7be48013"}
Jul 07 15:23:02.605293 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.605246    2560 scope.go:117] "RemoveContainer" containerID="b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6"
Jul 07 15:23:02.613766 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.613749    2560 scope.go:117] "RemoveContainer" containerID="b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6"
Jul 07 15:23:02.614004 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:23:02.613985    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6\": container with ID starting with b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6 not found: ID does not exist" containerID="b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6"
Jul 07 15:23:02.614049 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.614012    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6"} err="failed to get container status \"b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6\": rpc error: code = NotFound desc = could not find container \"b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6\": container with ID starting with b029084afa1328ceddfb1988e1c829b87474d14c8258f41843d450bd967340d6 not found: ID does not exist"
Jul 07 15:23:02.629349 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.629319    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-65vtm"]
Jul 07 15:23:02.639481 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:02.639455    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-65vtm"]
Jul 07 15:23:03.136161 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:03.136125    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59307d3-ad78-4a5e-a43f-e8176f9629af" path="/var/lib/kubelet/pods/b59307d3-ad78-4a5e-a43f-e8176f9629af/volumes"
Jul 07 15:23:04.930218 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:04.930184    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ch8m5"]
Jul 07 15:23:04.930685 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:04.930363    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-ch8m5" podUID="e0303433-4610-4ee5-bedf-d35c7c2e110d" containerName="authorino" containerID="cri-o://eec2c91f327cc450c57b345fa94da8ff9f176d6f9aedb0f1ba9ca4ad43512431" gracePeriod=30
Jul 07 15:23:05.618928 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:05.618892    2560 generic.go:356] "Generic (PLEG): container finished" podID="e0303433-4610-4ee5-bedf-d35c7c2e110d" containerID="eec2c91f327cc450c57b345fa94da8ff9f176d6f9aedb0f1ba9ca4ad43512431" exitCode=0
Jul 07 15:23:05.619122 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:05.618962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ch8m5" event={"ID":"e0303433-4610-4ee5-bedf-d35c7c2e110d","Type":"ContainerDied","Data":"eec2c91f327cc450c57b345fa94da8ff9f176d6f9aedb0f1ba9ca4ad43512431"}
Jul 07 15:23:05.730779 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:05.730502    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ch8m5"
Jul 07 15:23:05.769576 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:05.769542    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncw5s\" (UniqueName: \"kubernetes.io/projected/e0303433-4610-4ee5-bedf-d35c7c2e110d-kube-api-access-ncw5s\") pod \"e0303433-4610-4ee5-bedf-d35c7c2e110d\" (UID: \"e0303433-4610-4ee5-bedf-d35c7c2e110d\") "
Jul 07 15:23:05.771619 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:05.771594    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0303433-4610-4ee5-bedf-d35c7c2e110d-kube-api-access-ncw5s" (OuterVolumeSpecName: "kube-api-access-ncw5s") pod "e0303433-4610-4ee5-bedf-d35c7c2e110d" (UID: "e0303433-4610-4ee5-bedf-d35c7c2e110d"). InnerVolumeSpecName "kube-api-access-ncw5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:23:05.870378 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:05.870302    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncw5s\" (UniqueName: \"kubernetes.io/projected/e0303433-4610-4ee5-bedf-d35c7c2e110d-kube-api-access-ncw5s\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:23:06.625478 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:06.625451    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ch8m5"
Jul 07 15:23:06.625478 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:06.625473    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ch8m5" event={"ID":"e0303433-4610-4ee5-bedf-d35c7c2e110d","Type":"ContainerDied","Data":"fd48ce7659e4d08ddd8523607c2a790f2260b93e47bdbb452830960bf2b0d41d"}
Jul 07 15:23:06.626068 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:06.625563    2560 scope.go:117] "RemoveContainer" containerID="eec2c91f327cc450c57b345fa94da8ff9f176d6f9aedb0f1ba9ca4ad43512431"
Jul 07 15:23:06.649383 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:06.649350    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ch8m5"]
Jul 07 15:23:06.655122 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:06.655098    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ch8m5"]
Jul 07 15:23:07.136745 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:07.136707    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0303433-4610-4ee5-bedf-d35c7c2e110d" path="/var/lib/kubelet/pods/e0303433-4610-4ee5-bedf-d35c7c2e110d/volumes"
Jul 07 15:23:09.420202 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.420168    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f69c45b67-rzvf6"]
Jul 07 15:23:09.420726 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.420664    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59307d3-ad78-4a5e-a43f-e8176f9629af" containerName="authorino"
Jul 07 15:23:09.420726 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.420682    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59307d3-ad78-4a5e-a43f-e8176f9629af" containerName="authorino"
Jul 07 15:23:09.420726 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.420708    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0303433-4610-4ee5-bedf-d35c7c2e110d" containerName="authorino"
Jul 07 15:23:09.420726 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.420715    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0303433-4610-4ee5-bedf-d35c7c2e110d" containerName="authorino"
Jul 07 15:23:09.420935 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.420788    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59307d3-ad78-4a5e-a43f-e8176f9629af" containerName="authorino"
Jul 07 15:23:09.420935 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.420800    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0303433-4610-4ee5-bedf-d35c7c2e110d" containerName="authorino"
Jul 07 15:23:09.425350 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.425316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:09.428081 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.428057    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 15:23:09.428081 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.428064    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2rsxr\""
Jul 07 15:23:09.428251 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.428063    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 07 15:23:09.442638 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.442612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f69c45b67-rzvf6"]
Jul 07 15:23:09.501574 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.501538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert\") pod \"maas-controller-5f69c45b67-rzvf6\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") " pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:09.501762 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.501603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42pm\" (UniqueName: \"kubernetes.io/projected/fc2005ad-6529-4a75-b8b7-966e760cf6be-kube-api-access-x42pm\") pod \"maas-controller-5f69c45b67-rzvf6\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") " pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:09.602804 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.602768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42pm\" (UniqueName: \"kubernetes.io/projected/fc2005ad-6529-4a75-b8b7-966e760cf6be-kube-api-access-x42pm\") pod \"maas-controller-5f69c45b67-rzvf6\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") " pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:09.602983 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.602866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert\") pod \"maas-controller-5f69c45b67-rzvf6\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") " pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:09.603033 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:23:09.603001    2560 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 07 15:23:09.603068 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:23:09.603063    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert podName:fc2005ad-6529-4a75-b8b7-966e760cf6be nodeName:}" failed. No retries permitted until 2026-07-07 15:23:10.103046787 +0000 UTC m=+607.511155670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert") pod "maas-controller-5f69c45b67-rzvf6" (UID: "fc2005ad-6529-4a75-b8b7-966e760cf6be") : secret "maas-controller-webhook-cert" not found
Jul 07 15:23:09.614574 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.614546    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42pm\" (UniqueName: \"kubernetes.io/projected/fc2005ad-6529-4a75-b8b7-966e760cf6be-kube-api-access-x42pm\") pod \"maas-controller-5f69c45b67-rzvf6\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") " pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:09.626992 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.626958    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c85dbf754-qkgp4"]
Jul 07 15:23:09.629128 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.629108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:09.639740 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.639714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c85dbf754-qkgp4"]
Jul 07 15:23:09.703550 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.703447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9718de8a-c359-454c-bf65-1dad2e136158-webhook-cert\") pod \"maas-controller-c85dbf754-qkgp4\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") " pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:09.703739 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.703605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkcvw\" (UniqueName: \"kubernetes.io/projected/9718de8a-c359-454c-bf65-1dad2e136158-kube-api-access-nkcvw\") pod \"maas-controller-c85dbf754-qkgp4\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") " pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:09.804227 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.804192    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkcvw\" (UniqueName: \"kubernetes.io/projected/9718de8a-c359-454c-bf65-1dad2e136158-kube-api-access-nkcvw\") pod \"maas-controller-c85dbf754-qkgp4\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") " pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:09.804455 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.804249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9718de8a-c359-454c-bf65-1dad2e136158-webhook-cert\") pod \"maas-controller-c85dbf754-qkgp4\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") " pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:09.806852 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.806819    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9718de8a-c359-454c-bf65-1dad2e136158-webhook-cert\") pod \"maas-controller-c85dbf754-qkgp4\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") " pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:09.814164 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.814127    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkcvw\" (UniqueName: \"kubernetes.io/projected/9718de8a-c359-454c-bf65-1dad2e136158-kube-api-access-nkcvw\") pod \"maas-controller-c85dbf754-qkgp4\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") " pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:09.941188 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:09.941146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:10.076021 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:10.075988    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c85dbf754-qkgp4"]
Jul 07 15:23:10.078961 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:23:10.078933    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9718de8a_c359_454c_bf65_1dad2e136158.slice/crio-9ba64bfcedfc938fded6e63784506fb502226683f0dd52b8873064e79a678637 WatchSource:0}: Error finding container 9ba64bfcedfc938fded6e63784506fb502226683f0dd52b8873064e79a678637: Status 404 returned error can't find the container with id 9ba64bfcedfc938fded6e63784506fb502226683f0dd52b8873064e79a678637
Jul 07 15:23:10.106737 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:10.106704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert\") pod \"maas-controller-5f69c45b67-rzvf6\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") " pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:10.109024 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:10.109005    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert\") pod \"maas-controller-5f69c45b67-rzvf6\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") " pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:10.335899 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:10.335816    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:10.494599 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:10.494572    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f69c45b67-rzvf6"]
Jul 07 15:23:10.496996 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:23:10.496969    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2005ad_6529_4a75_b8b7_966e760cf6be.slice/crio-f08cf80685dbaa34e8af919146e03a29906feb4eda9222455c853bb025afee98 WatchSource:0}: Error finding container f08cf80685dbaa34e8af919146e03a29906feb4eda9222455c853bb025afee98: Status 404 returned error can't find the container with id f08cf80685dbaa34e8af919146e03a29906feb4eda9222455c853bb025afee98
Jul 07 15:23:10.645260 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:10.645225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c85dbf754-qkgp4" event={"ID":"9718de8a-c359-454c-bf65-1dad2e136158","Type":"ContainerStarted","Data":"9ba64bfcedfc938fded6e63784506fb502226683f0dd52b8873064e79a678637"}
Jul 07 15:23:10.646347 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:10.646324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f69c45b67-rzvf6" event={"ID":"fc2005ad-6529-4a75-b8b7-966e760cf6be","Type":"ContainerStarted","Data":"f08cf80685dbaa34e8af919146e03a29906feb4eda9222455c853bb025afee98"}
Jul 07 15:23:12.661748 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:12.661647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c85dbf754-qkgp4" event={"ID":"9718de8a-c359-454c-bf65-1dad2e136158","Type":"ContainerStarted","Data":"bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5"}
Jul 07 15:23:12.661748 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:12.661734    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:12.663170 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:12.663141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f69c45b67-rzvf6" event={"ID":"fc2005ad-6529-4a75-b8b7-966e760cf6be","Type":"ContainerStarted","Data":"63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4"}
Jul 07 15:23:12.663292 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:12.663269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:12.704146 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:12.704094    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c85dbf754-qkgp4" podStartSLOduration=1.38868883 podStartE2EDuration="3.704076027s" totalImagesPullingTime="2.315387197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:12.702711546 +0000 UTC m=+610.110820450" watchObservedRunningTime="2026-07-07 15:23:12.704076027 +0000 UTC m=+610.112184932"
Jul 07 15:23:12.725232 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:12.725171    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f69c45b67-rzvf6" podStartSLOduration=1.818588053 podStartE2EDuration="3.725152464s" totalImagesPullingTime="1.906564411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:12.724725862 +0000 UTC m=+610.132834764" watchObservedRunningTime="2026-07-07 15:23:12.725152464 +0000 UTC m=+610.133261366"
Jul 07 15:23:23.672833 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:23.672799    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:23.673241 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:23.672912    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:23:23.747909 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:23.747873    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f69c45b67-rzvf6"]
Jul 07 15:23:23.748087 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:23.748066    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5f69c45b67-rzvf6" podUID="fc2005ad-6529-4a75-b8b7-966e760cf6be" containerName="manager" containerID="cri-o://63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4" gracePeriod=30
Jul 07 15:23:23.998064 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:23.998037    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:24.129381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.129344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x42pm\" (UniqueName: \"kubernetes.io/projected/fc2005ad-6529-4a75-b8b7-966e760cf6be-kube-api-access-x42pm\") pod \"fc2005ad-6529-4a75-b8b7-966e760cf6be\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") "
Jul 07 15:23:24.129587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.129444    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert\") pod \"fc2005ad-6529-4a75-b8b7-966e760cf6be\" (UID: \"fc2005ad-6529-4a75-b8b7-966e760cf6be\") "
Jul 07 15:23:24.131613 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.131573    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fc2005ad-6529-4a75-b8b7-966e760cf6be" (UID: "fc2005ad-6529-4a75-b8b7-966e760cf6be"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:23:24.131733 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.131692    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2005ad-6529-4a75-b8b7-966e760cf6be-kube-api-access-x42pm" (OuterVolumeSpecName: "kube-api-access-x42pm") pod "fc2005ad-6529-4a75-b8b7-966e760cf6be" (UID: "fc2005ad-6529-4a75-b8b7-966e760cf6be"). InnerVolumeSpecName "kube-api-access-x42pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:23:24.229960 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.229929    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42pm\" (UniqueName: \"kubernetes.io/projected/fc2005ad-6529-4a75-b8b7-966e760cf6be-kube-api-access-x42pm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:23:24.229960 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.229954    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2005ad-6529-4a75-b8b7-966e760cf6be-webhook-cert\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:23:24.714214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.714179    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc2005ad-6529-4a75-b8b7-966e760cf6be" containerID="63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4" exitCode=0
Jul 07 15:23:24.714715 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.714234    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f69c45b67-rzvf6"
Jul 07 15:23:24.714715 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.714271    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f69c45b67-rzvf6" event={"ID":"fc2005ad-6529-4a75-b8b7-966e760cf6be","Type":"ContainerDied","Data":"63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4"}
Jul 07 15:23:24.714715 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.714329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f69c45b67-rzvf6" event={"ID":"fc2005ad-6529-4a75-b8b7-966e760cf6be","Type":"ContainerDied","Data":"f08cf80685dbaa34e8af919146e03a29906feb4eda9222455c853bb025afee98"}
Jul 07 15:23:24.714715 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.714354    2560 scope.go:117] "RemoveContainer" containerID="63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4"
Jul 07 15:23:24.722827 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.722813    2560 scope.go:117] "RemoveContainer" containerID="63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4"
Jul 07 15:23:24.723103 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:23:24.723086    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4\": container with ID starting with 63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4 not found: ID does not exist" containerID="63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4"
Jul 07 15:23:24.723145 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.723112    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4"} err="failed to get container status \"63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4\": rpc error: code = NotFound desc = could not find container \"63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4\": container with ID starting with 63ebb2a579683cf990a2bbba02adb76f536aef8b3f570aef3f08bf2a931f64e4 not found: ID does not exist"
Jul 07 15:23:24.738199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.738177    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f69c45b67-rzvf6"]
Jul 07 15:23:24.748219 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:24.748199    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f69c45b67-rzvf6"]
Jul 07 15:23:25.136360 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:25.136332    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2005ad-6529-4a75-b8b7-966e760cf6be" path="/var/lib/kubelet/pods/fc2005ad-6529-4a75-b8b7-966e760cf6be/volumes"
Jul 07 15:23:45.670893 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.670856    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5c8fbf54f8-6x7c5"]
Jul 07 15:23:45.671335 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.671322    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2005ad-6529-4a75-b8b7-966e760cf6be" containerName="manager"
Jul 07 15:23:45.671384 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.671337    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2005ad-6529-4a75-b8b7-966e760cf6be" containerName="manager"
Jul 07 15:23:45.671449 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.671423    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2005ad-6529-4a75-b8b7-966e760cf6be" containerName="manager"
Jul 07 15:23:45.674553 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.674536    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.677366 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.677346    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 07 15:23:45.677515 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.677370    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-2hd94\""
Jul 07 15:23:45.684709 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.684682    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5c8fbf54f8-6x7c5"]
Jul 07 15:23:45.706962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.706938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghsm\" (UniqueName: \"kubernetes.io/projected/6db1a320-c588-48e4-a74e-f3e2346ee645-kube-api-access-tghsm\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.707111 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.707014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db1a320-c588-48e4-a74e-f3e2346ee645-maas-api-tls\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.707111 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.707042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db1a320-c588-48e4-a74e-f3e2346ee645-openshift-service-ca\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.807737 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.807707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db1a320-c588-48e4-a74e-f3e2346ee645-maas-api-tls\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.807737 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.807739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db1a320-c588-48e4-a74e-f3e2346ee645-openshift-service-ca\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.807964 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.807786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghsm\" (UniqueName: \"kubernetes.io/projected/6db1a320-c588-48e4-a74e-f3e2346ee645-kube-api-access-tghsm\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.808513 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.808486    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db1a320-c588-48e4-a74e-f3e2346ee645-openshift-service-ca\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.810263 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.810241    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db1a320-c588-48e4-a74e-f3e2346ee645-maas-api-tls\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.824579 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.824550    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghsm\" (UniqueName: \"kubernetes.io/projected/6db1a320-c588-48e4-a74e-f3e2346ee645-kube-api-access-tghsm\") pod \"maas-api-5c8fbf54f8-6x7c5\" (UID: \"6db1a320-c588-48e4-a74e-f3e2346ee645\") " pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:45.985357 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:45.985276    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:46.122160 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.122069    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5c8fbf54f8-6x7c5"]
Jul 07 15:23:46.125026 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:23:46.124993    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db1a320_c588_48e4_a74e_f3e2346ee645.slice/crio-ab2ef36d227403f757b1300c23271b9829aed6330fe6e5191ab306146d3c16bd WatchSource:0}: Error finding container ab2ef36d227403f757b1300c23271b9829aed6330fe6e5191ab306146d3c16bd: Status 404 returned error can't find the container with id ab2ef36d227403f757b1300c23271b9829aed6330fe6e5191ab306146d3c16bd
Jul 07 15:23:46.183460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.183429    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9r8rt"]
Jul 07 15:23:46.186128 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.186108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.189149 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.189127    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 07 15:23:46.189563 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.189541    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gssfd\""
Jul 07 15:23:46.200465 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.200435    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9r8rt"]
Jul 07 15:23:46.210004 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.209976    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-5g7hl"]
Jul 07 15:23:46.210329 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.210296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c753c8-ff16-45fb-aa74-af8982415637-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9r8rt\" (UID: \"11c753c8-ff16-45fb-aa74-af8982415637\") " pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.210443 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.210331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhd8\" (UniqueName: \"kubernetes.io/projected/11c753c8-ff16-45fb-aa74-af8982415637-kube-api-access-wmhd8\") pod \"payload-pre-processing-7964457696-9r8rt\" (UID: \"11c753c8-ff16-45fb-aa74-af8982415637\") " pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.212310 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.212293    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.224001 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.223979    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-5g7hl"]
Jul 07 15:23:46.311629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.311546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c753c8-ff16-45fb-aa74-af8982415637-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9r8rt\" (UID: \"11c753c8-ff16-45fb-aa74-af8982415637\") " pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.311629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.311586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fsm8\" (UniqueName: \"kubernetes.io/projected/25bbea9b-4f7f-4e2d-ba7b-0064190894be-kube-api-access-6fsm8\") pod \"payload-processing-58f86c7798-5g7hl\" (UID: \"25bbea9b-4f7f-4e2d-ba7b-0064190894be\") " pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.311629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.311607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhd8\" (UniqueName: \"kubernetes.io/projected/11c753c8-ff16-45fb-aa74-af8982415637-kube-api-access-wmhd8\") pod \"payload-pre-processing-7964457696-9r8rt\" (UID: \"11c753c8-ff16-45fb-aa74-af8982415637\") " pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.311883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.311641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bbea9b-4f7f-4e2d-ba7b-0064190894be-plugins-config-volume\") pod \"payload-processing-58f86c7798-5g7hl\" (UID: \"25bbea9b-4f7f-4e2d-ba7b-0064190894be\") " pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.312282 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.312255    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c753c8-ff16-45fb-aa74-af8982415637-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9r8rt\" (UID: \"11c753c8-ff16-45fb-aa74-af8982415637\") " pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.319850 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.319824    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhd8\" (UniqueName: \"kubernetes.io/projected/11c753c8-ff16-45fb-aa74-af8982415637-kube-api-access-wmhd8\") pod \"payload-pre-processing-7964457696-9r8rt\" (UID: \"11c753c8-ff16-45fb-aa74-af8982415637\") " pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.412230 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.412198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fsm8\" (UniqueName: \"kubernetes.io/projected/25bbea9b-4f7f-4e2d-ba7b-0064190894be-kube-api-access-6fsm8\") pod \"payload-processing-58f86c7798-5g7hl\" (UID: \"25bbea9b-4f7f-4e2d-ba7b-0064190894be\") " pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.412433 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.412247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bbea9b-4f7f-4e2d-ba7b-0064190894be-plugins-config-volume\") pod \"payload-processing-58f86c7798-5g7hl\" (UID: \"25bbea9b-4f7f-4e2d-ba7b-0064190894be\") " pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.412950 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.412930    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bbea9b-4f7f-4e2d-ba7b-0064190894be-plugins-config-volume\") pod \"payload-processing-58f86c7798-5g7hl\" (UID: \"25bbea9b-4f7f-4e2d-ba7b-0064190894be\") " pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.424554 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.424530    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fsm8\" (UniqueName: \"kubernetes.io/projected/25bbea9b-4f7f-4e2d-ba7b-0064190894be-kube-api-access-6fsm8\") pod \"payload-processing-58f86c7798-5g7hl\" (UID: \"25bbea9b-4f7f-4e2d-ba7b-0064190894be\") " pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.497870 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.497842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:46.524140 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.524114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:46.642482 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.642446    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9r8rt"]
Jul 07 15:23:46.644821 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:23:46.644791    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c753c8_ff16_45fb_aa74_af8982415637.slice/crio-d6e264c495b2d4f32489b8d2011913ab9f159e2d0d92466772159665e2ae2838 WatchSource:0}: Error finding container d6e264c495b2d4f32489b8d2011913ab9f159e2d0d92466772159665e2ae2838: Status 404 returned error can't find the container with id d6e264c495b2d4f32489b8d2011913ab9f159e2d0d92466772159665e2ae2838
Jul 07 15:23:46.668112 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.668081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-5g7hl"]
Jul 07 15:23:46.670644 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:23:46.670620    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bbea9b_4f7f_4e2d_ba7b_0064190894be.slice/crio-0c060824a4eae56109fac721012f79c54e79574a0b577ccfc0d14d734dbc120b WatchSource:0}: Error finding container 0c060824a4eae56109fac721012f79c54e79574a0b577ccfc0d14d734dbc120b: Status 404 returned error can't find the container with id 0c060824a4eae56109fac721012f79c54e79574a0b577ccfc0d14d734dbc120b
Jul 07 15:23:46.806378 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.806342    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt" event={"ID":"11c753c8-ff16-45fb-aa74-af8982415637","Type":"ContainerStarted","Data":"d6e264c495b2d4f32489b8d2011913ab9f159e2d0d92466772159665e2ae2838"}
Jul 07 15:23:46.807533 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.807504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-5g7hl" event={"ID":"25bbea9b-4f7f-4e2d-ba7b-0064190894be","Type":"ContainerStarted","Data":"0c060824a4eae56109fac721012f79c54e79574a0b577ccfc0d14d734dbc120b"}
Jul 07 15:23:46.808569 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:46.808546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5c8fbf54f8-6x7c5" event={"ID":"6db1a320-c588-48e4-a74e-f3e2346ee645","Type":"ContainerStarted","Data":"ab2ef36d227403f757b1300c23271b9829aed6330fe6e5191ab306146d3c16bd"}
Jul 07 15:23:48.827244 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.827199    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt" event={"ID":"11c753c8-ff16-45fb-aa74-af8982415637","Type":"ContainerStarted","Data":"73dc8de9928b8f2bb6cfa762634fe98db8005c5d8cd277d220a8b22e2a1547dc"}
Jul 07 15:23:48.827728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.827299    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:48.829070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.829022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-5g7hl" event={"ID":"25bbea9b-4f7f-4e2d-ba7b-0064190894be","Type":"ContainerStarted","Data":"ee44dfefc1e6a8b0890b6fc90baf7740f6bf8857c6fa791cbfdbf6148b86d352"}
Jul 07 15:23:48.829209 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.829078    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:23:48.830534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.830513    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5c8fbf54f8-6x7c5" event={"ID":"6db1a320-c588-48e4-a74e-f3e2346ee645","Type":"ContainerStarted","Data":"7705d9c994b3b8f0dcbcc13680d6d4835634f2988b0049ca1bc0fb9083d67bd5"}
Jul 07 15:23:48.830669 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.830652    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:48.847475 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.847232    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt" podStartSLOduration=1.112599702 podStartE2EDuration="2.847213753s" totalImagesPullingTime="1.734614051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:48.845527324 +0000 UTC m=+646.253636208" watchObservedRunningTime="2026-07-07 15:23:48.847213753 +0000 UTC m=+646.255322655"
Jul 07 15:23:48.864730 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.864680    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-5g7hl" podStartSLOduration=1.152934875 podStartE2EDuration="2.864663581s" totalImagesPullingTime="1.711728706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:48.864092814 +0000 UTC m=+646.272201717" watchObservedRunningTime="2026-07-07 15:23:48.864663581 +0000 UTC m=+646.272772482"
Jul 07 15:23:48.886358 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:48.886312    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5c8fbf54f8-6x7c5" podStartSLOduration=1.630868881 podStartE2EDuration="3.886297452s" totalImagesPullingTime="2.255428571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:23:48.884581355 +0000 UTC m=+646.292690270" watchObservedRunningTime="2026-07-07 15:23:48.886297452 +0000 UTC m=+646.294406354"
Jul 07 15:23:54.841054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:54.841026    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5c8fbf54f8-6x7c5"
Jul 07 15:23:59.837602 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:59.837570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-9r8rt"
Jul 07 15:23:59.838005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:23:59.837647    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-5g7hl"
Jul 07 15:24:05.036880 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.036845    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"]
Jul 07 15:24:05.041582 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.041564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.044984 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.044919    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6kqtt\""
Jul 07 15:24:05.044984 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.044919    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 07 15:24:05.044984 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.044966    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 15:24:05.045196 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.044993    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 15:24:05.052617 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.052595    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"]
Jul 07 15:24:05.176837 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.176799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0381424-fe57-40ed-8cdf-852666730b54-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.176837 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.176836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.177056 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.176855    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.177056 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.176960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.177056 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.177018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8br\" (UniqueName: \"kubernetes.io/projected/f0381424-fe57-40ed-8cdf-852666730b54-kube-api-access-lw8br\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.177056 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.177049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278070 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8br\" (UniqueName: \"kubernetes.io/projected/f0381424-fe57-40ed-8cdf-852666730b54-kube-api-access-lw8br\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278288 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278288 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0381424-fe57-40ed-8cdf-852666730b54-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278426 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278426 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278558 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278536    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278624 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278554    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.278773 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.278752    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.280505 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.280488    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0381424-fe57-40ed-8cdf-852666730b54-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.280555 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.280544    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0381424-fe57-40ed-8cdf-852666730b54-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.287036 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.286980    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8br\" (UniqueName: \"kubernetes.io/projected/f0381424-fe57-40ed-8cdf-852666730b54-kube-api-access-lw8br\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk\" (UID: \"f0381424-fe57-40ed-8cdf-852666730b54\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.351500 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.351461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:05.488438 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.488400    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"]
Jul 07 15:24:05.491413 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:24:05.491374    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0381424_fe57_40ed_8cdf_852666730b54.slice/crio-e144965e7be522619642dfaffb66e36b9919c7ab0a6e2718a30cffbc17d03713 WatchSource:0}: Error finding container e144965e7be522619642dfaffb66e36b9919c7ab0a6e2718a30cffbc17d03713: Status 404 returned error can't find the container with id e144965e7be522619642dfaffb66e36b9919c7ab0a6e2718a30cffbc17d03713
Jul 07 15:24:05.492848 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.492829    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:24:05.905184 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:05.905109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk" event={"ID":"f0381424-fe57-40ed-8cdf-852666730b54","Type":"ContainerStarted","Data":"e144965e7be522619642dfaffb66e36b9919c7ab0a6e2718a30cffbc17d03713"}
Jul 07 15:24:06.910952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:06.910915    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk" event={"ID":"f0381424-fe57-40ed-8cdf-852666730b54","Type":"ContainerStarted","Data":"d0430a5241666bf95b2e615810e790983825d76fc6e73f3eaf1cbb76a4564f8f"}
Jul 07 15:24:06.931347 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:06.931293    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk" podStartSLOduration=0.685953629 podStartE2EDuration="1.931275602s" totalImagesPullingTime="1.245321973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:24:06.930661829 +0000 UTC m=+664.338770737" watchObservedRunningTime="2026-07-07 15:24:06.931275602 +0000 UTC m=+664.339384504"
Jul 07 15:24:15.352377 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:15.352337    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:15.352930 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:15.352442    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:15.364976 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:15.364948    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:15.962212 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:15.962181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk"
Jul 07 15:24:35.766302 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.766262    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"]
Jul 07 15:24:35.769680 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.769662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.773588 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.773566    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 07 15:24:35.789713 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.789687    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"]
Jul 07 15:24:35.848505 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.848467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe436ea-1efe-4841-8e72-b67cb132e2a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.848505 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.848507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.848722 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.848535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.848722 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.848607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.848722 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.848655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/7fe436ea-1efe-4841-8e72-b67cb132e2a4-kube-api-access-wj2bm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.848722 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.848708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.949888 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.949848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe436ea-1efe-4841-8e72-b67cb132e2a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.949888 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.949886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.950133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.949905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.950133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.949939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.950133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.949981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/7fe436ea-1efe-4841-8e72-b67cb132e2a4-kube-api-access-wj2bm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.950133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.950009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.950336 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.950244    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.950441 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.950419    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.950487 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.950436    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.952226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.952199    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe436ea-1efe-4841-8e72-b67cb132e2a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.952640 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.952621    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe436ea-1efe-4841-8e72-b67cb132e2a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:35.959889 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:35.959870    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2bm\" (UniqueName: \"kubernetes.io/projected/7fe436ea-1efe-4841-8e72-b67cb132e2a4-kube-api-access-wj2bm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d\" (UID: \"7fe436ea-1efe-4841-8e72-b67cb132e2a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:36.080687 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:36.080591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:36.215124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:36.215096    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"]
Jul 07 15:24:36.215844 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:24:36.215817    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe436ea_1efe_4841_8e72_b67cb132e2a4.slice/crio-e74475b4bc2167c3a41734437338ef4c5cd218b444445a10607a0667a0054811 WatchSource:0}: Error finding container e74475b4bc2167c3a41734437338ef4c5cd218b444445a10607a0667a0054811: Status 404 returned error can't find the container with id e74475b4bc2167c3a41734437338ef4c5cd218b444445a10607a0667a0054811
Jul 07 15:24:37.041367 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:37.041326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d" event={"ID":"7fe436ea-1efe-4841-8e72-b67cb132e2a4","Type":"ContainerStarted","Data":"8e36f4afd4f967ed0277c101b97cebbf62681649719aade55209be7301b04ea8"}
Jul 07 15:24:37.041875 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:37.041380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d" event={"ID":"7fe436ea-1efe-4841-8e72-b67cb132e2a4","Type":"ContainerStarted","Data":"e74475b4bc2167c3a41734437338ef4c5cd218b444445a10607a0667a0054811"}
Jul 07 15:24:37.063079 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:37.063025    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d" podStartSLOduration=1.675734974 podStartE2EDuration="2.063007903s" totalImagesPullingTime="387.272929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:24:37.061630866 +0000 UTC m=+694.469739779" watchObservedRunningTime="2026-07-07 15:24:37.063007903 +0000 UTC m=+694.471116804"
Jul 07 15:24:46.080821 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:46.080790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:46.081255 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:46.080860    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:46.093227 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:46.093198    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:24:47.098977 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:24:47.098940    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d"
Jul 07 15:25:03.934913 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:03.934871    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"]
Jul 07 15:25:03.943365 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:03.943335    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:03.946900 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:03.946872    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 15:25:03.948101 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:03.948076    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"]
Jul 07 15:25:04.102419 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.102358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.102600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.102427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.102600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.102488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxd9\" (UniqueName: \"kubernetes.io/projected/a00c2e63-b090-4886-90c8-81d96e52ef44-kube-api-access-mfxd9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.102600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.102546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.102600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.102577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.102600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.102620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a00c2e63-b090-4886-90c8-81d96e52ef44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203482 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203482 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203482 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxd9\" (UniqueName: \"kubernetes.io/projected/a00c2e63-b090-4886-90c8-81d96e52ef44-kube-api-access-mfxd9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a00c2e63-b090-4886-90c8-81d96e52ef44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203917 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203850    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203998 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203977    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.203998 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.203990    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.205846 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.205821    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a00c2e63-b090-4886-90c8-81d96e52ef44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.206059 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.206043    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a00c2e63-b090-4886-90c8-81d96e52ef44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.212669 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.212649    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxd9\" (UniqueName: \"kubernetes.io/projected/a00c2e63-b090-4886-90c8-81d96e52ef44-kube-api-access-mfxd9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz\" (UID: \"a00c2e63-b090-4886-90c8-81d96e52ef44\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.253106 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.253073    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:04.392791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:04.392764    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"]
Jul 07 15:25:04.394740 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:25:04.394713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda00c2e63_b090_4886_90c8_81d96e52ef44.slice/crio-760225a5b88908cdf7c1c5a9dfe841f561a6ea942809402d9fc277ff77a0e80f WatchSource:0}: Error finding container 760225a5b88908cdf7c1c5a9dfe841f561a6ea942809402d9fc277ff77a0e80f: Status 404 returned error can't find the container with id 760225a5b88908cdf7c1c5a9dfe841f561a6ea942809402d9fc277ff77a0e80f
Jul 07 15:25:05.171413 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:05.171359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz" event={"ID":"a00c2e63-b090-4886-90c8-81d96e52ef44","Type":"ContainerStarted","Data":"d5195de3ed35251c919cb0f1de5a8c2697de917f3583cde90b447dd49508d403"}
Jul 07 15:25:05.171794 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:05.171424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz" event={"ID":"a00c2e63-b090-4886-90c8-81d96e52ef44","Type":"ContainerStarted","Data":"760225a5b88908cdf7c1c5a9dfe841f561a6ea942809402d9fc277ff77a0e80f"}
Jul 07 15:25:05.192468 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:05.192415    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz" podStartSLOduration=1.915794262 podStartE2EDuration="2.192378764s" totalImagesPullingTime="276.584502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:25:05.190243285 +0000 UTC m=+722.598352209" watchObservedRunningTime="2026-07-07 15:25:05.192378764 +0000 UTC m=+722.600487665"
Jul 07 15:25:14.253420 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:14.253376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:14.253804 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:14.253439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:14.266057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:14.266033    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:25:15.224839 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:25:15.224810    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz"
Jul 07 15:27:17.825903 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:17.825862    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c85dbf754-qkgp4"]
Jul 07 15:27:17.826417 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:17.826120    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-c85dbf754-qkgp4" podUID="9718de8a-c359-454c-bf65-1dad2e136158" containerName="manager" containerID="cri-o://bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5" gracePeriod=30
Jul 07 15:27:18.073177 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.073152    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:27:18.183604 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.183576    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkcvw\" (UniqueName: \"kubernetes.io/projected/9718de8a-c359-454c-bf65-1dad2e136158-kube-api-access-nkcvw\") pod \"9718de8a-c359-454c-bf65-1dad2e136158\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") "
Jul 07 15:27:18.183808 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.183612    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9718de8a-c359-454c-bf65-1dad2e136158-webhook-cert\") pod \"9718de8a-c359-454c-bf65-1dad2e136158\" (UID: \"9718de8a-c359-454c-bf65-1dad2e136158\") "
Jul 07 15:27:18.185769 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.185741    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9718de8a-c359-454c-bf65-1dad2e136158-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9718de8a-c359-454c-bf65-1dad2e136158" (UID: "9718de8a-c359-454c-bf65-1dad2e136158"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:27:18.185883 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.185776    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9718de8a-c359-454c-bf65-1dad2e136158-kube-api-access-nkcvw" (OuterVolumeSpecName: "kube-api-access-nkcvw") pod "9718de8a-c359-454c-bf65-1dad2e136158" (UID: "9718de8a-c359-454c-bf65-1dad2e136158"). InnerVolumeSpecName "kube-api-access-nkcvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:27:18.284417 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.284358    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkcvw\" (UniqueName: \"kubernetes.io/projected/9718de8a-c359-454c-bf65-1dad2e136158-kube-api-access-nkcvw\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:27:18.284417 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.284412    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9718de8a-c359-454c-bf65-1dad2e136158-webhook-cert\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:27:18.750909 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.750868    2560 generic.go:356] "Generic (PLEG): container finished" podID="9718de8a-c359-454c-bf65-1dad2e136158" containerID="bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5" exitCode=0
Jul 07 15:27:18.751085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.750914    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c85dbf754-qkgp4"
Jul 07 15:27:18.751085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.750953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c85dbf754-qkgp4" event={"ID":"9718de8a-c359-454c-bf65-1dad2e136158","Type":"ContainerDied","Data":"bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5"}
Jul 07 15:27:18.751085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.751002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c85dbf754-qkgp4" event={"ID":"9718de8a-c359-454c-bf65-1dad2e136158","Type":"ContainerDied","Data":"9ba64bfcedfc938fded6e63784506fb502226683f0dd52b8873064e79a678637"}
Jul 07 15:27:18.751085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.751025    2560 scope.go:117] "RemoveContainer" containerID="bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5"
Jul 07 15:27:18.759926 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.759909    2560 scope.go:117] "RemoveContainer" containerID="bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5"
Jul 07 15:27:18.760178 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:27:18.760158    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5\": container with ID starting with bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5 not found: ID does not exist" containerID="bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5"
Jul 07 15:27:18.760237 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.760187    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5"} err="failed to get container status \"bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5\": rpc error: code = NotFound desc = could not find container \"bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5\": container with ID starting with bc376ce44d9c56840fb7a58288015c6ef2db4dbdf585e1c704432e528401a1a5 not found: ID does not exist"
Jul 07 15:27:18.773327 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.773298    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c85dbf754-qkgp4"]
Jul 07 15:27:18.777410 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:18.777375    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c85dbf754-qkgp4"]
Jul 07 15:27:19.136048 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.136013    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9718de8a-c359-454c-bf65-1dad2e136158" path="/var/lib/kubelet/pods/9718de8a-c359-454c-bf65-1dad2e136158/volumes"
Jul 07 15:27:19.586433 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.586333    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c85dbf754-xr2dp"]
Jul 07 15:27:19.586888 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.586867    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9718de8a-c359-454c-bf65-1dad2e136158" containerName="manager"
Jul 07 15:27:19.586933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.586894    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9718de8a-c359-454c-bf65-1dad2e136158" containerName="manager"
Jul 07 15:27:19.587019 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.587000    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9718de8a-c359-454c-bf65-1dad2e136158" containerName="manager"
Jul 07 15:27:19.591333 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.591313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:19.593734 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.593706    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 15:27:19.595025 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.594978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2rsxr\""
Jul 07 15:27:19.595133 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.595024    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 07 15:27:19.600803 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.600782    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c85dbf754-xr2dp"]
Jul 07 15:27:19.695823 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.695787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcb6\" (UniqueName: \"kubernetes.io/projected/04b812df-e722-4036-9a45-1f29267c50fa-kube-api-access-phcb6\") pod \"maas-controller-c85dbf754-xr2dp\" (UID: \"04b812df-e722-4036-9a45-1f29267c50fa\") " pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:19.695994 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.695865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04b812df-e722-4036-9a45-1f29267c50fa-webhook-cert\") pod \"maas-controller-c85dbf754-xr2dp\" (UID: \"04b812df-e722-4036-9a45-1f29267c50fa\") " pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:19.796754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.796712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04b812df-e722-4036-9a45-1f29267c50fa-webhook-cert\") pod \"maas-controller-c85dbf754-xr2dp\" (UID: \"04b812df-e722-4036-9a45-1f29267c50fa\") " pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:19.796930 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.796769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcb6\" (UniqueName: \"kubernetes.io/projected/04b812df-e722-4036-9a45-1f29267c50fa-kube-api-access-phcb6\") pod \"maas-controller-c85dbf754-xr2dp\" (UID: \"04b812df-e722-4036-9a45-1f29267c50fa\") " pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:19.799194 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.799177    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04b812df-e722-4036-9a45-1f29267c50fa-webhook-cert\") pod \"maas-controller-c85dbf754-xr2dp\" (UID: \"04b812df-e722-4036-9a45-1f29267c50fa\") " pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:19.806375 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.806349    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcb6\" (UniqueName: \"kubernetes.io/projected/04b812df-e722-4036-9a45-1f29267c50fa-kube-api-access-phcb6\") pod \"maas-controller-c85dbf754-xr2dp\" (UID: \"04b812df-e722-4036-9a45-1f29267c50fa\") " pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:19.902682 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:19.902645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:20.043728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:20.043703    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c85dbf754-xr2dp"]
Jul 07 15:27:20.045842 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:27:20.045813    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b812df_e722_4036_9a45_1f29267c50fa.slice/crio-e3dd9c92bd47bdfe4365ff4e418151d349ea6178243adffa681748d07e9d56c0 WatchSource:0}: Error finding container e3dd9c92bd47bdfe4365ff4e418151d349ea6178243adffa681748d07e9d56c0: Status 404 returned error can't find the container with id e3dd9c92bd47bdfe4365ff4e418151d349ea6178243adffa681748d07e9d56c0
Jul 07 15:27:20.761547 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:20.761447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c85dbf754-xr2dp" event={"ID":"04b812df-e722-4036-9a45-1f29267c50fa","Type":"ContainerStarted","Data":"0f103957f70a81b457d24ec7add25a8ec5fc267ba93075f5bf4adcc74ef3e373"}
Jul 07 15:27:20.761547 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:20.761540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c85dbf754-xr2dp" event={"ID":"04b812df-e722-4036-9a45-1f29267c50fa","Type":"ContainerStarted","Data":"e3dd9c92bd47bdfe4365ff4e418151d349ea6178243adffa681748d07e9d56c0"}
Jul 07 15:27:20.761985 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:20.761608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:27:20.782297 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:20.782248    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c85dbf754-xr2dp" podStartSLOduration=1.357685364 podStartE2EDuration="1.782231783s" totalImagesPullingTime="424.546419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:27:20.780467345 +0000 UTC m=+858.188576241" watchObservedRunningTime="2026-07-07 15:27:20.782231783 +0000 UTC m=+858.190340688"
Jul 07 15:27:31.772440 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:27:31.772408    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c85dbf754-xr2dp"
Jul 07 15:39:18.922079 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:18.922041    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"]
Jul 07 15:39:18.925905 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:18.925883    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:18.929159 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:18.929136    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-e7011447-serving-cert\""
Jul 07 15:39:18.946118 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:18.946095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"]
Jul 07 15:39:19.013507 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.013477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-maas-api-tls\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.013691 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.013560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-openshift-service-ca\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.013691 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.013597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hvg\" (UniqueName: \"kubernetes.io/projected/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-kube-api-access-z7hvg\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.114467 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.114435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-maas-api-tls\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.114648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.114509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-openshift-service-ca\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.114648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.114550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hvg\" (UniqueName: \"kubernetes.io/projected/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-kube-api-access-z7hvg\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.115257 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.115226    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-openshift-service-ca\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.116852 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.116834    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-maas-api-tls\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.126043 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.126022    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hvg\" (UniqueName: \"kubernetes.io/projected/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-kube-api-access-z7hvg\") pod \"maas-api-e2e-ait-e7011447-d7f5c747d-pqb55\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") " pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.237258 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.237173    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:19.376384 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.376359    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"]
Jul 07 15:39:19.378924 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:39:19.378901    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb815b2d_f24f_4fcb_ac45_ac26a9658e6c.slice/crio-3501106ac45fca9e4b92125e44d53ddb704c389b8e1fb3778bc13de151743027 WatchSource:0}: Error finding container 3501106ac45fca9e4b92125e44d53ddb704c389b8e1fb3778bc13de151743027: Status 404 returned error can't find the container with id 3501106ac45fca9e4b92125e44d53ddb704c389b8e1fb3778bc13de151743027
Jul 07 15:39:19.380633 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.380612    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:39:19.922088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:19.922046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55" event={"ID":"db815b2d-f24f-4fcb-ac45-ac26a9658e6c","Type":"ContainerStarted","Data":"3501106ac45fca9e4b92125e44d53ddb704c389b8e1fb3778bc13de151743027"}
Jul 07 15:39:20.928709 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:20.928670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55" event={"ID":"db815b2d-f24f-4fcb-ac45-ac26a9658e6c","Type":"ContainerStarted","Data":"2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4"}
Jul 07 15:39:20.929129 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:20.928829    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:20.951673 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:20.951621    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55" podStartSLOduration=2.453483605 podStartE2EDuration="2.951602771s" totalImagesPullingTime="498.119166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:39:20.951272883 +0000 UTC m=+1578.359381787" watchObservedRunningTime="2026-07-07 15:39:20.951602771 +0000 UTC m=+1578.359711672"
Jul 07 15:39:24.419305 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.419270    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"]
Jul 07 15:39:24.419795 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.419485    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55" podUID="db815b2d-f24f-4fcb-ac45-ac26a9658e6c" containerName="maas-api" containerID="cri-o://2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4" gracePeriod=30
Jul 07 15:39:24.424459 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.424436    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:24.671435 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.671362    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:24.868942 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.868905    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7hvg\" (UniqueName: \"kubernetes.io/projected/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-kube-api-access-z7hvg\") pod \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") "
Jul 07 15:39:24.869118 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.868986    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-openshift-service-ca\") pod \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") "
Jul 07 15:39:24.869118 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.869028    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-maas-api-tls\") pod \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\" (UID: \"db815b2d-f24f-4fcb-ac45-ac26a9658e6c\") "
Jul 07 15:39:24.869406 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.869361    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db815b2d-f24f-4fcb-ac45-ac26a9658e6c" (UID: "db815b2d-f24f-4fcb-ac45-ac26a9658e6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:39:24.871106 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.871081    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-kube-api-access-z7hvg" (OuterVolumeSpecName: "kube-api-access-z7hvg") pod "db815b2d-f24f-4fcb-ac45-ac26a9658e6c" (UID: "db815b2d-f24f-4fcb-ac45-ac26a9658e6c"). InnerVolumeSpecName "kube-api-access-z7hvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:39:24.871206 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.871113    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db815b2d-f24f-4fcb-ac45-ac26a9658e6c" (UID: "db815b2d-f24f-4fcb-ac45-ac26a9658e6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:39:24.948775 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.948689    2560 generic.go:356] "Generic (PLEG): container finished" podID="db815b2d-f24f-4fcb-ac45-ac26a9658e6c" containerID="2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4" exitCode=0
Jul 07 15:39:24.948775 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.948729    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"
Jul 07 15:39:24.948952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.948773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55" event={"ID":"db815b2d-f24f-4fcb-ac45-ac26a9658e6c","Type":"ContainerDied","Data":"2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4"}
Jul 07 15:39:24.948952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.948829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55" event={"ID":"db815b2d-f24f-4fcb-ac45-ac26a9658e6c","Type":"ContainerDied","Data":"3501106ac45fca9e4b92125e44d53ddb704c389b8e1fb3778bc13de151743027"}
Jul 07 15:39:24.948952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.948857    2560 scope.go:117] "RemoveContainer" containerID="2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4"
Jul 07 15:39:24.957592 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.957570    2560 scope.go:117] "RemoveContainer" containerID="2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4"
Jul 07 15:39:24.957883 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:39:24.957864    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4\": container with ID starting with 2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4 not found: ID does not exist" containerID="2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4"
Jul 07 15:39:24.957927 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.957894    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4"} err="failed to get container status \"2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4\": rpc error: code = NotFound desc = could not find container \"2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4\": container with ID starting with 2f5ae2f4f0d3e90e0392d302642f2b412adad34dc7b24eb158ef843964af7dc4 not found: ID does not exist"
Jul 07 15:39:24.970295 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.970267    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:24.970295 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.970298    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hvg\" (UniqueName: \"kubernetes.io/projected/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-kube-api-access-z7hvg\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:24.970537 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.970312    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db815b2d-f24f-4fcb-ac45-ac26a9658e6c-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:24.972347 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.972297    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"]
Jul 07 15:39:24.973927 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:24.973909    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-e7011447-d7f5c747d-pqb55"]
Jul 07 15:39:25.136644 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:25.136611    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db815b2d-f24f-4fcb-ac45-ac26a9658e6c" path="/var/lib/kubelet/pods/db815b2d-f24f-4fcb-ac45-ac26a9658e6c/volumes"
Jul 07 15:39:32.499052 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.499014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"]
Jul 07 15:39:32.499476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.499374    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db815b2d-f24f-4fcb-ac45-ac26a9658e6c" containerName="maas-api"
Jul 07 15:39:32.499476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.499405    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="db815b2d-f24f-4fcb-ac45-ac26a9658e6c" containerName="maas-api"
Jul 07 15:39:32.499564 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.499489    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="db815b2d-f24f-4fcb-ac45-ac26a9658e6c" containerName="maas-api"
Jul 07 15:39:32.502620 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.502600    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.504931 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.504912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-908066ab-serving-cert\""
Jul 07 15:39:32.515058 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.515033    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"]
Jul 07 15:39:32.535658 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.532592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a171ea9b-6e45-4f53-97ed-94c1e30030f4-openshift-service-ca\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.535658 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.532687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.535658 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.532745    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrzs\" (UniqueName: \"kubernetes.io/projected/a171ea9b-6e45-4f53-97ed-94c1e30030f4-kube-api-access-jsrzs\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.634205 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.634169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a171ea9b-6e45-4f53-97ed-94c1e30030f4-openshift-service-ca\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.634422 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.634248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.634422 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.634299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrzs\" (UniqueName: \"kubernetes.io/projected/a171ea9b-6e45-4f53-97ed-94c1e30030f4-kube-api-access-jsrzs\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.634541 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:39:32.634434    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-908066ab-serving-cert: secret "maas-api-e2e-ait-908066ab-serving-cert" not found
Jul 07 15:39:32.634541 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:39:32.634516    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls podName:a171ea9b-6e45-4f53-97ed-94c1e30030f4 nodeName:}" failed. No retries permitted until 2026-07-07 15:39:33.134496995 +0000 UTC m=+1590.542605877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls") pod "maas-api-e2e-ait-908066ab-7c7886456f-p9jpk" (UID: "a171ea9b-6e45-4f53-97ed-94c1e30030f4") : secret "maas-api-e2e-ait-908066ab-serving-cert" not found
Jul 07 15:39:32.634961 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.634941    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a171ea9b-6e45-4f53-97ed-94c1e30030f4-openshift-service-ca\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:32.645032 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:32.645004    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrzs\" (UniqueName: \"kubernetes.io/projected/a171ea9b-6e45-4f53-97ed-94c1e30030f4-kube-api-access-jsrzs\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:33.138263 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:33.138230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:33.140823 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:33.140789    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls\") pod \"maas-api-e2e-ait-908066ab-7c7886456f-p9jpk\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") " pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:33.415525 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:33.415425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:33.558599 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:39:33.558569    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda171ea9b_6e45_4f53_97ed_94c1e30030f4.slice/crio-79261b551028789f3c5675deb08f7cd8b26ea2c7bacddb8d46b0e28948b6a28a WatchSource:0}: Error finding container 79261b551028789f3c5675deb08f7cd8b26ea2c7bacddb8d46b0e28948b6a28a: Status 404 returned error can't find the container with id 79261b551028789f3c5675deb08f7cd8b26ea2c7bacddb8d46b0e28948b6a28a
Jul 07 15:39:33.559501 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:33.559462    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"]
Jul 07 15:39:33.991742 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:33.991704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk" event={"ID":"a171ea9b-6e45-4f53-97ed-94c1e30030f4","Type":"ContainerStarted","Data":"795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e"}
Jul 07 15:39:33.991918 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:33.991760    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk" event={"ID":"a171ea9b-6e45-4f53-97ed-94c1e30030f4","Type":"ContainerStarted","Data":"79261b551028789f3c5675deb08f7cd8b26ea2c7bacddb8d46b0e28948b6a28a"}
Jul 07 15:39:33.991918 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:33.991841    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:34.015889 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:34.015837    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk" podStartSLOduration=1.665446065 podStartE2EDuration="2.01581998s" totalImagesPullingTime="350.373915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:39:34.013848159 +0000 UTC m=+1591.421957060" watchObservedRunningTime="2026-07-07 15:39:34.01581998 +0000 UTC m=+1591.423928882"
Jul 07 15:39:38.030741 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.030705    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"]
Jul 07 15:39:38.031210 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.030988    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk" podUID="a171ea9b-6e45-4f53-97ed-94c1e30030f4" containerName="maas-api" containerID="cri-o://795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e" gracePeriod=30
Jul 07 15:39:38.036012 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.035991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:38.272118 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.272094    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:38.273997 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.273977    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsrzs\" (UniqueName: \"kubernetes.io/projected/a171ea9b-6e45-4f53-97ed-94c1e30030f4-kube-api-access-jsrzs\") pod \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") "
Jul 07 15:39:38.274077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.274033    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls\") pod \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") "
Jul 07 15:39:38.274077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.274063    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a171ea9b-6e45-4f53-97ed-94c1e30030f4-openshift-service-ca\") pod \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\" (UID: \"a171ea9b-6e45-4f53-97ed-94c1e30030f4\") "
Jul 07 15:39:38.274470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.274447    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a171ea9b-6e45-4f53-97ed-94c1e30030f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a171ea9b-6e45-4f53-97ed-94c1e30030f4" (UID: "a171ea9b-6e45-4f53-97ed-94c1e30030f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:39:38.275917 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.275888    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a171ea9b-6e45-4f53-97ed-94c1e30030f4-kube-api-access-jsrzs" (OuterVolumeSpecName: "kube-api-access-jsrzs") pod "a171ea9b-6e45-4f53-97ed-94c1e30030f4" (UID: "a171ea9b-6e45-4f53-97ed-94c1e30030f4"). InnerVolumeSpecName "kube-api-access-jsrzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:39:38.276068 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.276046    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a171ea9b-6e45-4f53-97ed-94c1e30030f4" (UID: "a171ea9b-6e45-4f53-97ed-94c1e30030f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:39:38.375485 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.375374    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a171ea9b-6e45-4f53-97ed-94c1e30030f4-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:38.375485 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.375433    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a171ea9b-6e45-4f53-97ed-94c1e30030f4-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:38.375485 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:38.375446    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrzs\" (UniqueName: \"kubernetes.io/projected/a171ea9b-6e45-4f53-97ed-94c1e30030f4-kube-api-access-jsrzs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:39.014556 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.014516    2560 generic.go:356] "Generic (PLEG): container finished" podID="a171ea9b-6e45-4f53-97ed-94c1e30030f4" containerID="795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e" exitCode=0
Jul 07 15:39:39.014766 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.014570    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"
Jul 07 15:39:39.014766 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.014610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk" event={"ID":"a171ea9b-6e45-4f53-97ed-94c1e30030f4","Type":"ContainerDied","Data":"795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e"}
Jul 07 15:39:39.014766 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.014675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk" event={"ID":"a171ea9b-6e45-4f53-97ed-94c1e30030f4","Type":"ContainerDied","Data":"79261b551028789f3c5675deb08f7cd8b26ea2c7bacddb8d46b0e28948b6a28a"}
Jul 07 15:39:39.014766 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.014705    2560 scope.go:117] "RemoveContainer" containerID="795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e"
Jul 07 15:39:39.024325 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.024308    2560 scope.go:117] "RemoveContainer" containerID="795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e"
Jul 07 15:39:39.024643 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:39:39.024620    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e\": container with ID starting with 795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e not found: ID does not exist" containerID="795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e"
Jul 07 15:39:39.024733 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.024652    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e"} err="failed to get container status \"795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e\": rpc error: code = NotFound desc = could not find container \"795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e\": container with ID starting with 795bf6e7ba5c54dca187cf712c2050fece9259cab19b23808deaae77bae3fb6e not found: ID does not exist"
Jul 07 15:39:39.046754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.046729    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"]
Jul 07 15:39:39.061602 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.061582    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-908066ab-7c7886456f-p9jpk"]
Jul 07 15:39:39.137026 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:39.136993    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a171ea9b-6e45-4f53-97ed-94c1e30030f4" path="/var/lib/kubelet/pods/a171ea9b-6e45-4f53-97ed-94c1e30030f4/volumes"
Jul 07 15:39:51.750680 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.750646    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"]
Jul 07 15:39:51.751044 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.751018    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a171ea9b-6e45-4f53-97ed-94c1e30030f4" containerName="maas-api"
Jul 07 15:39:51.751044 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.751030    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a171ea9b-6e45-4f53-97ed-94c1e30030f4" containerName="maas-api"
Jul 07 15:39:51.751112 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.751100    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a171ea9b-6e45-4f53-97ed-94c1e30030f4" containerName="maas-api"
Jul 07 15:39:51.760516 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.760481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.765379 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.765352    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-2b24ca6c-serving-cert\""
Jul 07 15:39:51.767937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.767873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"]
Jul 07 15:39:51.793456 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.793414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5d818-5c11-49ec-8dbb-85494e17a943-openshift-service-ca\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.793635 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.793465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7nf\" (UniqueName: \"kubernetes.io/projected/c6d5d818-5c11-49ec-8dbb-85494e17a943-kube-api-access-5f7nf\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.793635 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.793502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.894725 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.894681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.894931 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.894785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5d818-5c11-49ec-8dbb-85494e17a943-openshift-service-ca\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.894931 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.894822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7nf\" (UniqueName: \"kubernetes.io/projected/c6d5d818-5c11-49ec-8dbb-85494e17a943-kube-api-access-5f7nf\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.894931 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:39:51.894846    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-2b24ca6c-serving-cert: secret "maas-api-e2e-derive-2b24ca6c-serving-cert" not found
Jul 07 15:39:51.895106 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:39:51.894942    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls podName:c6d5d818-5c11-49ec-8dbb-85494e17a943 nodeName:}" failed. No retries permitted until 2026-07-07 15:39:52.394919018 +0000 UTC m=+1609.803027919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls") pod "maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm" (UID: "c6d5d818-5c11-49ec-8dbb-85494e17a943") : secret "maas-api-e2e-derive-2b24ca6c-serving-cert" not found
Jul 07 15:39:51.895511 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.895490    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5d818-5c11-49ec-8dbb-85494e17a943-openshift-service-ca\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:51.911084 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:51.911051    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7nf\" (UniqueName: \"kubernetes.io/projected/c6d5d818-5c11-49ec-8dbb-85494e17a943-kube-api-access-5f7nf\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:52.399714 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:52.399673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:52.402072 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:52.402051    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls\") pod \"maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") " pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:52.674738 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:52.674646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:52.819428 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:52.819365    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"]
Jul 07 15:39:52.822616 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:39:52.822588    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d5d818_5c11_49ec_8dbb_85494e17a943.slice/crio-b6761de3a3c1a794f236483e0963692e8fabdc2d2439547eb0fcec4e1fb1a8a8 WatchSource:0}: Error finding container b6761de3a3c1a794f236483e0963692e8fabdc2d2439547eb0fcec4e1fb1a8a8: Status 404 returned error can't find the container with id b6761de3a3c1a794f236483e0963692e8fabdc2d2439547eb0fcec4e1fb1a8a8
Jul 07 15:39:53.081265 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:53.081176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm" event={"ID":"c6d5d818-5c11-49ec-8dbb-85494e17a943","Type":"ContainerStarted","Data":"b6761de3a3c1a794f236483e0963692e8fabdc2d2439547eb0fcec4e1fb1a8a8"}
Jul 07 15:39:54.086648 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:54.086553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm" event={"ID":"c6d5d818-5c11-49ec-8dbb-85494e17a943","Type":"ContainerStarted","Data":"d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e"}
Jul 07 15:39:54.087018 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:54.086652    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:54.109811 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:54.109757    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm" podStartSLOduration=2.669557801 podStartE2EDuration="3.109738975s" totalImagesPullingTime="440.181174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:39:54.107144466 +0000 UTC m=+1611.515253371" watchObservedRunningTime="2026-07-07 15:39:54.109738975 +0000 UTC m=+1611.517847961"
Jul 07 15:39:56.635216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.635181    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"]
Jul 07 15:39:56.636091 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.635433    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm" podUID="c6d5d818-5c11-49ec-8dbb-85494e17a943" containerName="maas-api" containerID="cri-o://d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e" gracePeriod=30
Jul 07 15:39:56.884199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.884173    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:56.940343 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.940262    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f7nf\" (UniqueName: \"kubernetes.io/projected/c6d5d818-5c11-49ec-8dbb-85494e17a943-kube-api-access-5f7nf\") pod \"c6d5d818-5c11-49ec-8dbb-85494e17a943\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") "
Jul 07 15:39:56.940343 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.940324    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5d818-5c11-49ec-8dbb-85494e17a943-openshift-service-ca\") pod \"c6d5d818-5c11-49ec-8dbb-85494e17a943\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") "
Jul 07 15:39:56.940600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.940412    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls\") pod \"c6d5d818-5c11-49ec-8dbb-85494e17a943\" (UID: \"c6d5d818-5c11-49ec-8dbb-85494e17a943\") "
Jul 07 15:39:56.940742 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.940719    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d5d818-5c11-49ec-8dbb-85494e17a943-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6d5d818-5c11-49ec-8dbb-85494e17a943" (UID: "c6d5d818-5c11-49ec-8dbb-85494e17a943"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:39:56.942315 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.942293    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6d5d818-5c11-49ec-8dbb-85494e17a943" (UID: "c6d5d818-5c11-49ec-8dbb-85494e17a943"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:39:56.942461 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:56.942346    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d5d818-5c11-49ec-8dbb-85494e17a943-kube-api-access-5f7nf" (OuterVolumeSpecName: "kube-api-access-5f7nf") pod "c6d5d818-5c11-49ec-8dbb-85494e17a943" (UID: "c6d5d818-5c11-49ec-8dbb-85494e17a943"). InnerVolumeSpecName "kube-api-access-5f7nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:39:57.041791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.041753    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d5d818-5c11-49ec-8dbb-85494e17a943-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:57.041791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.041787    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7nf\" (UniqueName: \"kubernetes.io/projected/c6d5d818-5c11-49ec-8dbb-85494e17a943-kube-api-access-5f7nf\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:57.041791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.041796    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d5d818-5c11-49ec-8dbb-85494e17a943-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:39:57.101828 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.101794    2560 generic.go:356] "Generic (PLEG): container finished" podID="c6d5d818-5c11-49ec-8dbb-85494e17a943" containerID="d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e" exitCode=0
Jul 07 15:39:57.101994 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.101833    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"
Jul 07 15:39:57.101994 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.101868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm" event={"ID":"c6d5d818-5c11-49ec-8dbb-85494e17a943","Type":"ContainerDied","Data":"d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e"}
Jul 07 15:39:57.101994 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.101922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm" event={"ID":"c6d5d818-5c11-49ec-8dbb-85494e17a943","Type":"ContainerDied","Data":"b6761de3a3c1a794f236483e0963692e8fabdc2d2439547eb0fcec4e1fb1a8a8"}
Jul 07 15:39:57.101994 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.101944    2560 scope.go:117] "RemoveContainer" containerID="d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e"
Jul 07 15:39:57.110639 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.110621    2560 scope.go:117] "RemoveContainer" containerID="d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e"
Jul 07 15:39:57.110898 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:39:57.110875    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e\": container with ID starting with d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e not found: ID does not exist" containerID="d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e"
Jul 07 15:39:57.110959 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.110910    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e"} err="failed to get container status \"d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e\": rpc error: code = NotFound desc = could not find container \"d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e\": container with ID starting with d6ff79a8ebae8ff69cd6b4ca65750723f2ce6c7d78d778678591beb857d0e01e not found: ID does not exist"
Jul 07 15:39:57.137916 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.137877    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"]
Jul 07 15:39:57.137916 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:57.137920    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-2b24ca6c-65b77c64f-lkqwm"]
Jul 07 15:39:59.136592 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:39:59.136556    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d5d818-5c11-49ec-8dbb-85494e17a943" path="/var/lib/kubelet/pods/c6d5d818-5c11-49ec-8dbb-85494e17a943/volumes"
Jul 07 15:41:45.657340 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.657304    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"]
Jul 07 15:41:45.659838 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.657691    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d5d818-5c11-49ec-8dbb-85494e17a943" containerName="maas-api"
Jul 07 15:41:45.659838 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.657706    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d5d818-5c11-49ec-8dbb-85494e17a943" containerName="maas-api"
Jul 07 15:41:45.659838 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.657772    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d5d818-5c11-49ec-8dbb-85494e17a943" containerName="maas-api"
Jul 07 15:41:45.660752 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.660734    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.663852 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.663825    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-77bbe860-serving-cert\""
Jul 07 15:41:45.679226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.679197    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"]
Jul 07 15:41:45.805266 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.805230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d63e-d178-49ef-8321-1edab4685951-openshift-service-ca\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.805469 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.805297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqrgd\" (UniqueName: \"kubernetes.io/projected/5ce9d63e-d178-49ef-8321-1edab4685951-kube-api-access-sqrgd\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.805469 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.805411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce9d63e-d178-49ef-8321-1edab4685951-maas-api-tls\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.906538 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.906501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqrgd\" (UniqueName: \"kubernetes.io/projected/5ce9d63e-d178-49ef-8321-1edab4685951-kube-api-access-sqrgd\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.906709 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.906548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce9d63e-d178-49ef-8321-1edab4685951-maas-api-tls\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.906709 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.906602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d63e-d178-49ef-8321-1edab4685951-openshift-service-ca\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.907181 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.907163    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d63e-d178-49ef-8321-1edab4685951-openshift-service-ca\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.909040 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.908995    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce9d63e-d178-49ef-8321-1edab4685951-maas-api-tls\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.918728 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.918704    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqrgd\" (UniqueName: \"kubernetes.io/projected/5ce9d63e-d178-49ef-8321-1edab4685951-kube-api-access-sqrgd\") pod \"maas-api-e2e-mt-77bbe860-569fc64767-mbbcc\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") " pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:45.972916 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:45.972891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:46.114799 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:46.114768    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"]
Jul 07 15:41:46.117862 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:41:46.117826    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce9d63e_d178_49ef_8321_1edab4685951.slice/crio-2827edc9b87c8890b19a3b970d215f33652da7764d1da02f2bc18ca6f92957d9 WatchSource:0}: Error finding container 2827edc9b87c8890b19a3b970d215f33652da7764d1da02f2bc18ca6f92957d9: Status 404 returned error can't find the container with id 2827edc9b87c8890b19a3b970d215f33652da7764d1da02f2bc18ca6f92957d9
Jul 07 15:41:46.578075 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:46.578029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc" event={"ID":"5ce9d63e-d178-49ef-8321-1edab4685951","Type":"ContainerStarted","Data":"2827edc9b87c8890b19a3b970d215f33652da7764d1da02f2bc18ca6f92957d9"}
Jul 07 15:41:47.584312 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:47.584274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc" event={"ID":"5ce9d63e-d178-49ef-8321-1edab4685951","Type":"ContainerStarted","Data":"f1c3345667277bd0dfd9b96c57fcd4fe092628d028510aa11cc01cdd10a50fb9"}
Jul 07 15:41:47.584732 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:47.584420    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:47.612019 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:47.611962    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc" podStartSLOduration=2.150956912 podStartE2EDuration="2.611947586s" totalImagesPullingTime="460.990674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:47.609042977 +0000 UTC m=+1725.017151880" watchObservedRunningTime="2026-07-07 15:41:47.611947586 +0000 UTC m=+1725.020056528"
Jul 07 15:41:53.593722 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:53.593693    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:58.439452 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.439411    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"]
Jul 07 15:41:58.439915 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.439720    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc" podUID="5ce9d63e-d178-49ef-8321-1edab4685951" containerName="maas-api" containerID="cri-o://f1c3345667277bd0dfd9b96c57fcd4fe092628d028510aa11cc01cdd10a50fb9" gracePeriod=30
Jul 07 15:41:58.589503 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.589460    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc" podUID="5ce9d63e-d178-49ef-8321-1edab4685951" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.40:8443/health\": dial tcp 10.134.0.40:8443: connect: connection refused"
Jul 07 15:41:58.632735 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.632702    2560 generic.go:356] "Generic (PLEG): container finished" podID="5ce9d63e-d178-49ef-8321-1edab4685951" containerID="f1c3345667277bd0dfd9b96c57fcd4fe092628d028510aa11cc01cdd10a50fb9" exitCode=0
Jul 07 15:41:58.633023 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.632752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc" event={"ID":"5ce9d63e-d178-49ef-8321-1edab4685951","Type":"ContainerDied","Data":"f1c3345667277bd0dfd9b96c57fcd4fe092628d028510aa11cc01cdd10a50fb9"}
Jul 07 15:41:58.689063 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.689041    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:58.726631 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.726548    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce9d63e-d178-49ef-8321-1edab4685951-maas-api-tls\") pod \"5ce9d63e-d178-49ef-8321-1edab4685951\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") "
Jul 07 15:41:58.726631 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.726618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d63e-d178-49ef-8321-1edab4685951-openshift-service-ca\") pod \"5ce9d63e-d178-49ef-8321-1edab4685951\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") "
Jul 07 15:41:58.726851 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.726647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqrgd\" (UniqueName: \"kubernetes.io/projected/5ce9d63e-d178-49ef-8321-1edab4685951-kube-api-access-sqrgd\") pod \"5ce9d63e-d178-49ef-8321-1edab4685951\" (UID: \"5ce9d63e-d178-49ef-8321-1edab4685951\") "
Jul 07 15:41:58.726971 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.726941    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce9d63e-d178-49ef-8321-1edab4685951-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ce9d63e-d178-49ef-8321-1edab4685951" (UID: "5ce9d63e-d178-49ef-8321-1edab4685951"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:41:58.728718 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.728697    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce9d63e-d178-49ef-8321-1edab4685951-kube-api-access-sqrgd" (OuterVolumeSpecName: "kube-api-access-sqrgd") pod "5ce9d63e-d178-49ef-8321-1edab4685951" (UID: "5ce9d63e-d178-49ef-8321-1edab4685951"). InnerVolumeSpecName "kube-api-access-sqrgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:41:58.728897 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.728871    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce9d63e-d178-49ef-8321-1edab4685951-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ce9d63e-d178-49ef-8321-1edab4685951" (UID: "5ce9d63e-d178-49ef-8321-1edab4685951"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:41:58.827765 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.827731    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce9d63e-d178-49ef-8321-1edab4685951-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:41:58.827765 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.827757    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d63e-d178-49ef-8321-1edab4685951-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:41:58.827765 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:58.827770    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqrgd\" (UniqueName: \"kubernetes.io/projected/5ce9d63e-d178-49ef-8321-1edab4685951-kube-api-access-sqrgd\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:41:59.640317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:59.640293    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"
Jul 07 15:41:59.640794 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:59.640296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc" event={"ID":"5ce9d63e-d178-49ef-8321-1edab4685951","Type":"ContainerDied","Data":"2827edc9b87c8890b19a3b970d215f33652da7764d1da02f2bc18ca6f92957d9"}
Jul 07 15:41:59.640794 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:59.640447    2560 scope.go:117] "RemoveContainer" containerID="f1c3345667277bd0dfd9b96c57fcd4fe092628d028510aa11cc01cdd10a50fb9"
Jul 07 15:41:59.663267 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:59.663240    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"]
Jul 07 15:41:59.667531 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:41:59.667507    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-77bbe860-569fc64767-mbbcc"]
Jul 07 15:42:01.136341 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:01.136308    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ce9d63e-d178-49ef-8321-1edab4685951" path="/var/lib/kubelet/pods/5ce9d63e-d178-49ef-8321-1edab4685951/volumes"
Jul 07 15:42:29.551968 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.551935    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"]
Jul 07 15:42:29.552596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.552443    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce9d63e-d178-49ef-8321-1edab4685951" containerName="maas-api"
Jul 07 15:42:29.552596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.552461    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce9d63e-d178-49ef-8321-1edab4685951" containerName="maas-api"
Jul 07 15:42:29.552596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.552570    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ce9d63e-d178-49ef-8321-1edab4685951" containerName="maas-api"
Jul 07 15:42:29.555537 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.555518    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.557937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.557916    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-73736f44-serving-cert\""
Jul 07 15:42:29.570982 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.570955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"]
Jul 07 15:42:29.651756 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.651707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.651756 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.651744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrtb\" (UniqueName: \"kubernetes.io/projected/30bd47f5-6724-450b-ae39-956f3af5ee54-kube-api-access-kmrtb\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.651978 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.651786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bd47f5-6724-450b-ae39-956f3af5ee54-openshift-service-ca\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.752720 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.752683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bd47f5-6724-450b-ae39-956f3af5ee54-openshift-service-ca\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.752901 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.752771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.752901 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.752791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrtb\" (UniqueName: \"kubernetes.io/projected/30bd47f5-6724-450b-ae39-956f3af5ee54-kube-api-access-kmrtb\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.752990 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:42:29.752929    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-73736f44-serving-cert: secret "maas-api-e2e-mt-73736f44-serving-cert" not found
Jul 07 15:42:29.753026 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:42:29.752993    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls podName:30bd47f5-6724-450b-ae39-956f3af5ee54 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:30.252974784 +0000 UTC m=+1767.661083666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls") pod "maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7" (UID: "30bd47f5-6724-450b-ae39-956f3af5ee54") : secret "maas-api-e2e-mt-73736f44-serving-cert" not found
Jul 07 15:42:29.753566 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.753541    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bd47f5-6724-450b-ae39-956f3af5ee54-openshift-service-ca\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:29.768047 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:29.768017    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrtb\" (UniqueName: \"kubernetes.io/projected/30bd47f5-6724-450b-ae39-956f3af5ee54-kube-api-access-kmrtb\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:30.257060 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:30.257029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:30.259542 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:30.259520    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls\") pod \"maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") " pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:30.465566 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:30.465529    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:30.611623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:30.611597    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"]
Jul 07 15:42:30.614280 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:42:30.614255    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bd47f5_6724_450b_ae39_956f3af5ee54.slice/crio-09044dff154881cd07463c28a2995393ec718268269dcc1d100cd16cd1c874fe WatchSource:0}: Error finding container 09044dff154881cd07463c28a2995393ec718268269dcc1d100cd16cd1c874fe: Status 404 returned error can't find the container with id 09044dff154881cd07463c28a2995393ec718268269dcc1d100cd16cd1c874fe
Jul 07 15:42:30.779088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:30.779004    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7" event={"ID":"30bd47f5-6724-450b-ae39-956f3af5ee54","Type":"ContainerStarted","Data":"09044dff154881cd07463c28a2995393ec718268269dcc1d100cd16cd1c874fe"}
Jul 07 15:42:31.785761 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:31.785727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7" event={"ID":"30bd47f5-6724-450b-ae39-956f3af5ee54","Type":"ContainerStarted","Data":"ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99"}
Jul 07 15:42:31.786226 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:31.785839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:31.811418 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:31.811343    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7" podStartSLOduration=2.273390724 podStartE2EDuration="2.811323152s" totalImagesPullingTime="537.932428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:31.808762069 +0000 UTC m=+1769.216870971" watchObservedRunningTime="2026-07-07 15:42:31.811323152 +0000 UTC m=+1769.219432060"
Jul 07 15:42:37.796450 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:37.796421    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:42:41.637309 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.637271    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"]
Jul 07 15:42:41.640591 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.640576    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.644008 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.643984    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73736f44\"/\"default-dockercfg-q8t86\""
Jul 07 15:42:41.644146 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.644034    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73736f44\"/\"openshift-service-ca.crt\""
Jul 07 15:42:41.644146 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.644053    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73736f44\"/\"e2e-lifecycle-model-73736f44-kserve-self-signed-certs\""
Jul 07 15:42:41.644146 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.644076    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73736f44\"/\"kube-root-ca.crt\""
Jul 07 15:42:41.652452 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.652430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"]
Jul 07 15:42:41.769057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.769024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1674b-61aa-4dd2-955c-8920460200e9-tls-certs\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.769057 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.769064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2f7p\" (UniqueName: \"kubernetes.io/projected/36b1674b-61aa-4dd2-955c-8920460200e9-kube-api-access-j2f7p\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.769266 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.769090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-tmp-dir\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.769266 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.769148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-model-cache\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.769266 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.769180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-dshm\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.769266 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.769215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-home\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.869694 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.869656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1674b-61aa-4dd2-955c-8920460200e9-tls-certs\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.869694 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.869691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f7p\" (UniqueName: \"kubernetes.io/projected/36b1674b-61aa-4dd2-955c-8920460200e9-kube-api-access-j2f7p\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.869951 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.869713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-tmp-dir\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.869951 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.869738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-model-cache\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.869951 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.869937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-dshm\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.870103 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.869995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-home\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.870199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.870170    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-tmp-dir\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.870199 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.870191    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-model-cache\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.870362 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.870341    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-home\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.872015 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.871997    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-dshm\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.872265 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.872248    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1674b-61aa-4dd2-955c-8920460200e9-tls-certs\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.878683 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.878662    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f7p\" (UniqueName: \"kubernetes.io/projected/36b1674b-61aa-4dd2-955c-8920460200e9-kube-api-access-j2f7p\") pod \"e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") " pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:41.949740 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:41.949648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:42.090108 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:42.090074    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"]
Jul 07 15:42:42.093904 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:42:42.093873    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36b1674b_61aa_4dd2_955c_8920460200e9.slice/crio-fd232806b4087e544bcc7dc5370ebef1b614f54caa98e34b5c9a4f27f764ea02 WatchSource:0}: Error finding container fd232806b4087e544bcc7dc5370ebef1b614f54caa98e34b5c9a4f27f764ea02: Status 404 returned error can't find the container with id fd232806b4087e544bcc7dc5370ebef1b614f54caa98e34b5c9a4f27f764ea02
Jul 07 15:42:42.836198 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:42.836162    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578" event={"ID":"36b1674b-61aa-4dd2-955c-8920460200e9","Type":"ContainerStarted","Data":"9a5af7f7a2377bc7f7425d9ec90e43b42bc83424739051f3153c51d5dfcdf9a4"}
Jul 07 15:42:42.836595 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:42.836207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578" event={"ID":"36b1674b-61aa-4dd2-955c-8920460200e9","Type":"ContainerStarted","Data":"fd232806b4087e544bcc7dc5370ebef1b614f54caa98e34b5c9a4f27f764ea02"}
Jul 07 15:42:42.857164 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:42.857119    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578" podStartSLOduration=1.8571040239999999 podStartE2EDuration="1.857104024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:42.855201757 +0000 UTC m=+1780.263310659" watchObservedRunningTime="2026-07-07 15:42:42.857104024 +0000 UTC m=+1780.265212927"
Jul 07 15:42:51.950432 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:51.950377    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:51.950903 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:51.950448    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:51.963068 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:51.963042    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:42:52.891326 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:42:52.891297    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:43:17.184659 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.184623    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"]
Jul 07 15:43:17.185116 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.184886    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7" podUID="30bd47f5-6724-450b-ae39-956f3af5ee54" containerName="maas-api" containerID="cri-o://ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99" gracePeriod=30
Jul 07 15:43:17.433060 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.433034    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:43:17.501587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.501494    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bd47f5-6724-450b-ae39-956f3af5ee54-openshift-service-ca\") pod \"30bd47f5-6724-450b-ae39-956f3af5ee54\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") "
Jul 07 15:43:17.501587 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.501564    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmrtb\" (UniqueName: \"kubernetes.io/projected/30bd47f5-6724-450b-ae39-956f3af5ee54-kube-api-access-kmrtb\") pod \"30bd47f5-6724-450b-ae39-956f3af5ee54\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") "
Jul 07 15:43:17.501812 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.501592    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls\") pod \"30bd47f5-6724-450b-ae39-956f3af5ee54\" (UID: \"30bd47f5-6724-450b-ae39-956f3af5ee54\") "
Jul 07 15:43:17.501887 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.501865    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bd47f5-6724-450b-ae39-956f3af5ee54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30bd47f5-6724-450b-ae39-956f3af5ee54" (UID: "30bd47f5-6724-450b-ae39-956f3af5ee54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:43:17.503705 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.503671    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30bd47f5-6724-450b-ae39-956f3af5ee54" (UID: "30bd47f5-6724-450b-ae39-956f3af5ee54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:43:17.503824 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.503746    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bd47f5-6724-450b-ae39-956f3af5ee54-kube-api-access-kmrtb" (OuterVolumeSpecName: "kube-api-access-kmrtb") pod "30bd47f5-6724-450b-ae39-956f3af5ee54" (UID: "30bd47f5-6724-450b-ae39-956f3af5ee54"). InnerVolumeSpecName "kube-api-access-kmrtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:43:17.602791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.602755    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bd47f5-6724-450b-ae39-956f3af5ee54-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:17.602791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.602783    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrtb\" (UniqueName: \"kubernetes.io/projected/30bd47f5-6724-450b-ae39-956f3af5ee54-kube-api-access-kmrtb\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:17.602791 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.602794    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30bd47f5-6724-450b-ae39-956f3af5ee54-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:17.992963 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.992925    2560 generic.go:356] "Generic (PLEG): container finished" podID="30bd47f5-6724-450b-ae39-956f3af5ee54" containerID="ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99" exitCode=0
Jul 07 15:43:17.993178 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.992972    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"
Jul 07 15:43:17.993178 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.992982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7" event={"ID":"30bd47f5-6724-450b-ae39-956f3af5ee54","Type":"ContainerDied","Data":"ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99"}
Jul 07 15:43:17.993178 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.993032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7" event={"ID":"30bd47f5-6724-450b-ae39-956f3af5ee54","Type":"ContainerDied","Data":"09044dff154881cd07463c28a2995393ec718268269dcc1d100cd16cd1c874fe"}
Jul 07 15:43:17.993178 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:17.993066    2560 scope.go:117] "RemoveContainer" containerID="ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99"
Jul 07 15:43:18.002749 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:18.002727    2560 scope.go:117] "RemoveContainer" containerID="ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99"
Jul 07 15:43:18.002984 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:43:18.002968    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99\": container with ID starting with ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99 not found: ID does not exist" containerID="ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99"
Jul 07 15:43:18.003024 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:18.002993    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99"} err="failed to get container status \"ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99\": rpc error: code = NotFound desc = could not find container \"ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99\": container with ID starting with ac308c8371dc26408c0328ad5809b785cfd3af73e15d812572adc2ac86feed99 not found: ID does not exist"
Jul 07 15:43:18.026036 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:18.026008    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"]
Jul 07 15:43:18.038185 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:18.038162    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-73736f44-6f87b888c6-jvtx7"]
Jul 07 15:43:19.136870 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:19.136835    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30bd47f5-6724-450b-ae39-956f3af5ee54" path="/var/lib/kubelet/pods/30bd47f5-6724-450b-ae39-956f3af5ee54/volumes"
Jul 07 15:43:27.871881 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:27.871847    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"]
Jul 07 15:43:42.891899 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:42.891853    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578" podUID="36b1674b-61aa-4dd2-955c-8920460200e9" containerName="main" containerID="cri-o://9a5af7f7a2377bc7f7425d9ec90e43b42bc83424739051f3153c51d5dfcdf9a4" gracePeriod=45
Jul 07 15:43:43.115137 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.115079    2560 generic.go:356] "Generic (PLEG): container finished" podID="36b1674b-61aa-4dd2-955c-8920460200e9" containerID="9a5af7f7a2377bc7f7425d9ec90e43b42bc83424739051f3153c51d5dfcdf9a4" exitCode=0
Jul 07 15:43:43.115298 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.115152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578" event={"ID":"36b1674b-61aa-4dd2-955c-8920460200e9","Type":"ContainerDied","Data":"9a5af7f7a2377bc7f7425d9ec90e43b42bc83424739051f3153c51d5dfcdf9a4"}
Jul 07 15:43:43.138471 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.138450    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:43:43.232514 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232424    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-dshm\") pod \"36b1674b-61aa-4dd2-955c-8920460200e9\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") "
Jul 07 15:43:43.232514 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232473    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-home\") pod \"36b1674b-61aa-4dd2-955c-8920460200e9\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") "
Jul 07 15:43:43.232514 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232510    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1674b-61aa-4dd2-955c-8920460200e9-tls-certs\") pod \"36b1674b-61aa-4dd2-955c-8920460200e9\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") "
Jul 07 15:43:43.232795 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-model-cache\") pod \"36b1674b-61aa-4dd2-955c-8920460200e9\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") "
Jul 07 15:43:43.232795 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232594    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-tmp-dir\") pod \"36b1674b-61aa-4dd2-955c-8920460200e9\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") "
Jul 07 15:43:43.232795 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232657    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2f7p\" (UniqueName: \"kubernetes.io/projected/36b1674b-61aa-4dd2-955c-8920460200e9-kube-api-access-j2f7p\") pod \"36b1674b-61aa-4dd2-955c-8920460200e9\" (UID: \"36b1674b-61aa-4dd2-955c-8920460200e9\") "
Jul 07 15:43:43.232956 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232801    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-home" (OuterVolumeSpecName: "home") pod "36b1674b-61aa-4dd2-955c-8920460200e9" (UID: "36b1674b-61aa-4dd2-955c-8920460200e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:43:43.232956 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232843    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "36b1674b-61aa-4dd2-955c-8920460200e9" (UID: "36b1674b-61aa-4dd2-955c-8920460200e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:43:43.233033 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232969    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:43.233033 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.232990    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:43.233131 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.233075    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36b1674b-61aa-4dd2-955c-8920460200e9" (UID: "36b1674b-61aa-4dd2-955c-8920460200e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:43:43.234702 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.234676    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b1674b-61aa-4dd2-955c-8920460200e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36b1674b-61aa-4dd2-955c-8920460200e9" (UID: "36b1674b-61aa-4dd2-955c-8920460200e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:43:43.234835 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.234787    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-dshm" (OuterVolumeSpecName: "dshm") pod "36b1674b-61aa-4dd2-955c-8920460200e9" (UID: "36b1674b-61aa-4dd2-955c-8920460200e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:43:43.234835 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.234791    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b1674b-61aa-4dd2-955c-8920460200e9-kube-api-access-j2f7p" (OuterVolumeSpecName: "kube-api-access-j2f7p") pod "36b1674b-61aa-4dd2-955c-8920460200e9" (UID: "36b1674b-61aa-4dd2-955c-8920460200e9"). InnerVolumeSpecName "kube-api-access-j2f7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:43:43.334214 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.334178    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2f7p\" (UniqueName: \"kubernetes.io/projected/36b1674b-61aa-4dd2-955c-8920460200e9-kube-api-access-j2f7p\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:43.334381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.334209    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:43.334381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.334233    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1674b-61aa-4dd2-955c-8920460200e9-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:43.334381 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:43.334245    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1674b-61aa-4dd2-955c-8920460200e9-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:43:44.120663 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:44.120631    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"
Jul 07 15:43:44.120663 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:44.120640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578" event={"ID":"36b1674b-61aa-4dd2-955c-8920460200e9","Type":"ContainerDied","Data":"fd232806b4087e544bcc7dc5370ebef1b614f54caa98e34b5c9a4f27f764ea02"}
Jul 07 15:43:44.121173 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:44.120691    2560 scope.go:117] "RemoveContainer" containerID="9a5af7f7a2377bc7f7425d9ec90e43b42bc83424739051f3153c51d5dfcdf9a4"
Jul 07 15:43:44.142758 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:44.142728    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"]
Jul 07 15:43:44.146162 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:44.146135    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-73736f44/e2e-lifecycle-model-73736f44-kserve-85d58cc7f5-kt578"]
Jul 07 15:43:45.138102 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:43:45.138059    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b1674b-61aa-4dd2-955c-8920460200e9" path="/var/lib/kubelet/pods/36b1674b-61aa-4dd2-955c-8920460200e9/volumes"
Jul 07 15:45:11.291670 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.291629    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"]
Jul 07 15:45:11.292136 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.291963    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b1674b-61aa-4dd2-955c-8920460200e9" containerName="main"
Jul 07 15:45:11.292136 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.291974    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b1674b-61aa-4dd2-955c-8920460200e9" containerName="main"
Jul 07 15:45:11.292136 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.292007    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bd47f5-6724-450b-ae39-956f3af5ee54" containerName="maas-api"
Jul 07 15:45:11.292136 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.292013    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bd47f5-6724-450b-ae39-956f3af5ee54" containerName="maas-api"
Jul 07 15:45:11.292136 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.292067    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b1674b-61aa-4dd2-955c-8920460200e9" containerName="main"
Jul 07 15:45:11.292136 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.292077    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bd47f5-6724-450b-ae39-956f3af5ee54" containerName="maas-api"
Jul 07 15:45:11.294981 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.294961    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.297913 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.297885    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-99d0b0-serving-cert\""
Jul 07 15:45:11.311784 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.311757    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"]
Jul 07 15:45:11.378571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.378541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a71916-f2ef-40aa-adbb-f119fa568f4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.378768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.378661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.378768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.378717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxh7n\" (UniqueName: \"kubernetes.io/projected/b0a71916-f2ef-40aa-adbb-f119fa568f4d-kube-api-access-pxh7n\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.479785 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.479749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.479972 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.479801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxh7n\" (UniqueName: \"kubernetes.io/projected/b0a71916-f2ef-40aa-adbb-f119fa568f4d-kube-api-access-pxh7n\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.479972 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.479825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a71916-f2ef-40aa-adbb-f119fa568f4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.479972 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:45:11.479911    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-99d0b0-serving-cert: secret "maas-api-e2e-inf-a-99d0b0-serving-cert" not found
Jul 07 15:45:11.480100 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:45:11.480005    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls podName:b0a71916-f2ef-40aa-adbb-f119fa568f4d nodeName:}" failed. No retries permitted until 2026-07-07 15:45:11.979983584 +0000 UTC m=+1929.388092465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls") pod "maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph" (UID: "b0a71916-f2ef-40aa-adbb-f119fa568f4d") : secret "maas-api-e2e-inf-a-99d0b0-serving-cert" not found
Jul 07 15:45:11.480487 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.480469    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a71916-f2ef-40aa-adbb-f119fa568f4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.491170 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.491144    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxh7n\" (UniqueName: \"kubernetes.io/projected/b0a71916-f2ef-40aa-adbb-f119fa568f4d-kube-api-access-pxh7n\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.984238 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.984201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:11.986750 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:11.986721    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") " pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:12.205412 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:12.205362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:12.342527 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:12.342504    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"]
Jul 07 15:45:12.344535 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:45:12.344505    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a71916_f2ef_40aa_adbb_f119fa568f4d.slice/crio-d443dba1492ca1d40af58587aa6c1dd573b8488b2b68a249a210908bf7628eff WatchSource:0}: Error finding container d443dba1492ca1d40af58587aa6c1dd573b8488b2b68a249a210908bf7628eff: Status 404 returned error can't find the container with id d443dba1492ca1d40af58587aa6c1dd573b8488b2b68a249a210908bf7628eff
Jul 07 15:45:12.345937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:12.345920    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:45:12.510854 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:12.510767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph" event={"ID":"b0a71916-f2ef-40aa-adbb-f119fa568f4d","Type":"ContainerStarted","Data":"d443dba1492ca1d40af58587aa6c1dd573b8488b2b68a249a210908bf7628eff"}
Jul 07 15:45:14.520792 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:14.520746    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph" event={"ID":"b0a71916-f2ef-40aa-adbb-f119fa568f4d","Type":"ContainerStarted","Data":"6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666"}
Jul 07 15:45:14.521203 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:14.520857    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:14.540298 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:14.540247    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph" podStartSLOduration=2.322330231 podStartE2EDuration="3.540228861s" totalImagesPullingTime="1.21789863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:14.538957383 +0000 UTC m=+1931.947066288" watchObservedRunningTime="2026-07-07 15:45:14.540228861 +0000 UTC m=+1931.948337763"
Jul 07 15:45:19.290127 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.290091    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"]
Jul 07 15:45:19.293877 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.293859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.296496 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.296466    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-ffde9f-serving-cert\""
Jul 07 15:45:19.304356 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.304321    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"]
Jul 07 15:45:19.353615 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.353580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102ac0df-ffa2-415d-8bc5-be83f360b216-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.353796 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.353639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mfm6\" (UniqueName: \"kubernetes.io/projected/102ac0df-ffa2-415d-8bc5-be83f360b216-kube-api-access-9mfm6\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.353796 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.353696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.454526 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.454489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102ac0df-ffa2-415d-8bc5-be83f360b216-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.454735 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.454547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mfm6\" (UniqueName: \"kubernetes.io/projected/102ac0df-ffa2-415d-8bc5-be83f360b216-kube-api-access-9mfm6\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.454735 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.454568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.454735 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:45:19.454671    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-ffde9f-serving-cert: secret "maas-api-e2e-inf-b-ffde9f-serving-cert" not found
Jul 07 15:45:19.454735 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:45:19.454730    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls podName:102ac0df-ffa2-415d-8bc5-be83f360b216 nodeName:}" failed. No retries permitted until 2026-07-07 15:45:19.95471543 +0000 UTC m=+1937.362824311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls") pod "maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn" (UID: "102ac0df-ffa2-415d-8bc5-be83f360b216") : secret "maas-api-e2e-inf-b-ffde9f-serving-cert" not found
Jul 07 15:45:19.455238 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.455211    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102ac0df-ffa2-415d-8bc5-be83f360b216-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.465347 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.465317    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mfm6\" (UniqueName: \"kubernetes.io/projected/102ac0df-ffa2-415d-8bc5-be83f360b216-kube-api-access-9mfm6\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.959707 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.959669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:19.962115 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:19.962094    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls\") pod \"maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") " pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:20.205223 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:20.205186    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:20.342616 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:20.342543    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"]
Jul 07 15:45:20.345083 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:45:20.345055    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102ac0df_ffa2_415d_8bc5_be83f360b216.slice/crio-a949807c5d923836d9c2a4e7c285cc6ca04f418cd04442cd5090228abcea0284 WatchSource:0}: Error finding container a949807c5d923836d9c2a4e7c285cc6ca04f418cd04442cd5090228abcea0284: Status 404 returned error can't find the container with id a949807c5d923836d9c2a4e7c285cc6ca04f418cd04442cd5090228abcea0284
Jul 07 15:45:20.531614 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:20.531540    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:45:20.549265 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:20.549229    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn" event={"ID":"102ac0df-ffa2-415d-8bc5-be83f360b216","Type":"ContainerStarted","Data":"a949807c5d923836d9c2a4e7c285cc6ca04f418cd04442cd5090228abcea0284"}
Jul 07 15:45:21.555734 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:21.555696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn" event={"ID":"102ac0df-ffa2-415d-8bc5-be83f360b216","Type":"ContainerStarted","Data":"ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e"}
Jul 07 15:45:21.556168 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:21.555866    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:21.577680 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:21.577633    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn" podStartSLOduration=2.093269232 podStartE2EDuration="2.577617188s" totalImagesPullingTime="484.347956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:21.575479898 +0000 UTC m=+1938.983588801" watchObservedRunningTime="2026-07-07 15:45:21.577617188 +0000 UTC m=+1938.985726090"
Jul 07 15:45:27.565907 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:27.565874    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:45:39.228632 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.228600    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"]
Jul 07 15:45:39.232060 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.232036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.235829 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.235806    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-99d0b0\"/\"default-dockercfg-btgzx\""
Jul 07 15:45:39.235977 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.235832    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-99d0b0\"/\"test-model-99d0b0-kserve-self-signed-certs\""
Jul 07 15:45:39.235977 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.235848    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-99d0b0\"/\"kube-root-ca.crt\""
Jul 07 15:45:39.235977 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.235806    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-99d0b0\"/\"openshift-service-ca.crt\""
Jul 07 15:45:39.243412 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.243374    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"]
Jul 07 15:45:39.344944 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.344914    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-model-cache\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.344944 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.344946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-home\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.345160 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.344967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ktl\" (UniqueName: \"kubernetes.io/projected/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-kube-api-access-68ktl\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.345160 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.344993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tls-certs\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.345160 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.345098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-dshm\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.345160 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.345131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tmp-dir\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446210 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tmp-dir\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446418 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-model-cache\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446418 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-home\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446418 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ktl\" (UniqueName: \"kubernetes.io/projected/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-kube-api-access-68ktl\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446418 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tls-certs\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446418 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-dshm\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446688 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446638    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tmp-dir\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446688 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446673    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-model-cache\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.446754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.446710    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-home\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.448544 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.448526    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-dshm\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.448810 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.448794    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tls-certs\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.455210 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.455184    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ktl\" (UniqueName: \"kubernetes.io/projected/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-kube-api-access-68ktl\") pod \"test-model-99d0b0-kserve-8675cfc96f-n7hgb\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") " pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.544385 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.544301    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:39.688592 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:39.688570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"]
Jul 07 15:45:39.690580 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:45:39.690543    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bfd2407_f4a8_43c9_8994_58f0f6b254c4.slice/crio-89600011f8a92d647c0bd44d93f7c4020cac5fb79946bd7ae114fdd6d3a337b3 WatchSource:0}: Error finding container 89600011f8a92d647c0bd44d93f7c4020cac5fb79946bd7ae114fdd6d3a337b3: Status 404 returned error can't find the container with id 89600011f8a92d647c0bd44d93f7c4020cac5fb79946bd7ae114fdd6d3a337b3
Jul 07 15:45:40.649137 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:40.649102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb" event={"ID":"0bfd2407-f4a8-43c9-8994-58f0f6b254c4","Type":"ContainerStarted","Data":"be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5"}
Jul 07 15:45:40.649137 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:40.649142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb" event={"ID":"0bfd2407-f4a8-43c9-8994-58f0f6b254c4","Type":"ContainerStarted","Data":"89600011f8a92d647c0bd44d93f7c4020cac5fb79946bd7ae114fdd6d3a337b3"}
Jul 07 15:45:40.669541 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:40.669477    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb" podStartSLOduration=1.6694619670000002 podStartE2EDuration="1.669461967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:40.667524819 +0000 UTC m=+1958.075633722" watchObservedRunningTime="2026-07-07 15:45:40.669461967 +0000 UTC m=+1958.077570890"
Jul 07 15:45:49.545361 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:49.545319    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:49.545907 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:49.545380    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:49.557877 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:49.557849    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:45:49.701361 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:45:49.701333    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:46:07.533339 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.533305    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"]
Jul 07 15:46:07.537173 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.537151    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.539534 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.539510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffde9f\"/\"openshift-service-ca.crt\""
Jul 07 15:46:07.540461 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.540437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffde9f\"/\"test-model-ffde9f-kserve-self-signed-certs\""
Jul 07 15:46:07.540580 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.540488    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ffde9f\"/\"default-dockercfg-cc574\""
Jul 07 15:46:07.540580 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.540519    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ffde9f\"/\"kube-root-ca.crt\""
Jul 07 15:46:07.547072 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.547046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"]
Jul 07 15:46:07.600873 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.600832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-home\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.600873 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.600878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznwf\" (UniqueName: \"kubernetes.io/projected/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-kube-api-access-qznwf\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.601105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.600904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-dshm\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.601105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.600969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-model-cache\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.601105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.601025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tmp-dir\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.601105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.601043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tls-certs\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.701857 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.701816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-home\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.701857 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.701861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznwf\" (UniqueName: \"kubernetes.io/projected/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-kube-api-access-qznwf\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.702131 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.701898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-dshm\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.702131 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.701927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-model-cache\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.702131 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.701963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tmp-dir\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.702131 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.701981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tls-certs\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.702322 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.702300    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-home\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.702448 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.702427    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-model-cache\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.702497 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.702456    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tmp-dir\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.704315 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.704294    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-dshm\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.704642 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.704626    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tls-certs\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.712105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.712082    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznwf\" (UniqueName: \"kubernetes.io/projected/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-kube-api-access-qznwf\") pod \"test-model-ffde9f-kserve-79c8fdbff7-xxjgb\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") " pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.848743 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.848638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:07.987958 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:07.987891    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"]
Jul 07 15:46:07.989994 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:46:07.989969    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedcaa7c3_2a97_40d3_abf3_dc5701d6d1c1.slice/crio-5c5032ca85596fb2f1f2612f2c1eb0aeb2129f06050567cfdd248ede01dec3f0 WatchSource:0}: Error finding container 5c5032ca85596fb2f1f2612f2c1eb0aeb2129f06050567cfdd248ede01dec3f0: Status 404 returned error can't find the container with id 5c5032ca85596fb2f1f2612f2c1eb0aeb2129f06050567cfdd248ede01dec3f0
Jul 07 15:46:08.776891 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:08.776858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb" event={"ID":"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1","Type":"ContainerStarted","Data":"0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe"}
Jul 07 15:46:08.777288 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:08.776903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb" event={"ID":"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1","Type":"ContainerStarted","Data":"5c5032ca85596fb2f1f2612f2c1eb0aeb2129f06050567cfdd248ede01dec3f0"}
Jul 07 15:46:08.795917 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:08.795861    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb" podStartSLOduration=1.795843466 podStartE2EDuration="1.795843466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:46:08.794139814 +0000 UTC m=+1986.202248718" watchObservedRunningTime="2026-07-07 15:46:08.795843466 +0000 UTC m=+1986.203952367"
Jul 07 15:46:17.849517 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:17.849473    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:17.849517 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:17.849526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:17.862242 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:17.862216    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:18.832008 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:18.831978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:27.406410 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:27.402050    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"]
Jul 07 15:46:28.380580 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:28.380542    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"]
Jul 07 15:46:29.318363 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.318324    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"]
Jul 07 15:46:29.318803 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.318566    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph" podUID="b0a71916-f2ef-40aa-adbb-f119fa568f4d" containerName="maas-api" containerID="cri-o://6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666" gracePeriod=30
Jul 07 15:46:29.566058 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.566029    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:46:29.709423 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.709374    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxh7n\" (UniqueName: \"kubernetes.io/projected/b0a71916-f2ef-40aa-adbb-f119fa568f4d-kube-api-access-pxh7n\") pod \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") "
Jul 07 15:46:29.709623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.709447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls\") pod \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") "
Jul 07 15:46:29.709623 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.709516    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a71916-f2ef-40aa-adbb-f119fa568f4d-openshift-service-ca\") pod \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\" (UID: \"b0a71916-f2ef-40aa-adbb-f119fa568f4d\") "
Jul 07 15:46:29.709863 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.709841    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a71916-f2ef-40aa-adbb-f119fa568f4d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0a71916-f2ef-40aa-adbb-f119fa568f4d" (UID: "b0a71916-f2ef-40aa-adbb-f119fa568f4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:46:29.711448 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.711426    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a71916-f2ef-40aa-adbb-f119fa568f4d-kube-api-access-pxh7n" (OuterVolumeSpecName: "kube-api-access-pxh7n") pod "b0a71916-f2ef-40aa-adbb-f119fa568f4d" (UID: "b0a71916-f2ef-40aa-adbb-f119fa568f4d"). InnerVolumeSpecName "kube-api-access-pxh7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:46:29.711569 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.711488    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0a71916-f2ef-40aa-adbb-f119fa568f4d" (UID: "b0a71916-f2ef-40aa-adbb-f119fa568f4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:46:29.811115 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.811066    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a71916-f2ef-40aa-adbb-f119fa568f4d-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:29.811115 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.811111    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxh7n\" (UniqueName: \"kubernetes.io/projected/b0a71916-f2ef-40aa-adbb-f119fa568f4d-kube-api-access-pxh7n\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:29.811115 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.811121    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a71916-f2ef-40aa-adbb-f119fa568f4d-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:29.872321 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.872283    2560 generic.go:356] "Generic (PLEG): container finished" podID="b0a71916-f2ef-40aa-adbb-f119fa568f4d" containerID="6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666" exitCode=0
Jul 07 15:46:29.872514 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.872332    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"
Jul 07 15:46:29.872514 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.872366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph" event={"ID":"b0a71916-f2ef-40aa-adbb-f119fa568f4d","Type":"ContainerDied","Data":"6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666"}
Jul 07 15:46:29.872514 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.872448    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph" event={"ID":"b0a71916-f2ef-40aa-adbb-f119fa568f4d","Type":"ContainerDied","Data":"d443dba1492ca1d40af58587aa6c1dd573b8488b2b68a249a210908bf7628eff"}
Jul 07 15:46:29.872514 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.872474    2560 scope.go:117] "RemoveContainer" containerID="6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666"
Jul 07 15:46:29.881674 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.881655    2560 scope.go:117] "RemoveContainer" containerID="6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666"
Jul 07 15:46:29.881935 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:46:29.881912    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666\": container with ID starting with 6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666 not found: ID does not exist" containerID="6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666"
Jul 07 15:46:29.881987 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.881943    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666"} err="failed to get container status \"6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666\": rpc error: code = NotFound desc = could not find container \"6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666\": container with ID starting with 6155c07872b9dada5137c86ffd0a70a7fd615eab6e59378d4ffd816083b3e666 not found: ID does not exist"
Jul 07 15:46:29.894290 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.894264    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"]
Jul 07 15:46:29.899031 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:29.899005    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-99d0b0-7886b66448-n9cph"]
Jul 07 15:46:31.142245 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:31.142201    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a71916-f2ef-40aa-adbb-f119fa568f4d" path="/var/lib/kubelet/pods/b0a71916-f2ef-40aa-adbb-f119fa568f4d/volumes"
Jul 07 15:46:42.428129 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.428079    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb" podUID="0bfd2407-f4a8-43c9-8994-58f0f6b254c4" containerName="main" containerID="cri-o://be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5" gracePeriod=45
Jul 07 15:46:42.673372 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.673348    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:46:42.731096 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731002    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-dshm\") pod \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") "
Jul 07 15:46:42.731096 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731074    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68ktl\" (UniqueName: \"kubernetes.io/projected/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-kube-api-access-68ktl\") pod \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") "
Jul 07 15:46:42.731330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731128    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-model-cache\") pod \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") "
Jul 07 15:46:42.731330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731153    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tmp-dir\") pod \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") "
Jul 07 15:46:42.731330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731239    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-home\") pod \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") "
Jul 07 15:46:42.731330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731261    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tls-certs\") pod \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\" (UID: \"0bfd2407-f4a8-43c9-8994-58f0f6b254c4\") "
Jul 07 15:46:42.731556 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731454    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bfd2407-f4a8-43c9-8994-58f0f6b254c4" (UID: "0bfd2407-f4a8-43c9-8994-58f0f6b254c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:42.731556 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731514    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-home" (OuterVolumeSpecName: "home") pod "0bfd2407-f4a8-43c9-8994-58f0f6b254c4" (UID: "0bfd2407-f4a8-43c9-8994-58f0f6b254c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:42.731556 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731530    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bfd2407-f4a8-43c9-8994-58f0f6b254c4" (UID: "0bfd2407-f4a8-43c9-8994-58f0f6b254c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:42.731689 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731597    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:42.731689 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731613    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:42.731689 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.731621    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:42.733359 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.733337    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-dshm" (OuterVolumeSpecName: "dshm") pod "0bfd2407-f4a8-43c9-8994-58f0f6b254c4" (UID: "0bfd2407-f4a8-43c9-8994-58f0f6b254c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:42.733453 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.733363    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-kube-api-access-68ktl" (OuterVolumeSpecName: "kube-api-access-68ktl") pod "0bfd2407-f4a8-43c9-8994-58f0f6b254c4" (UID: "0bfd2407-f4a8-43c9-8994-58f0f6b254c4"). InnerVolumeSpecName "kube-api-access-68ktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:46:42.733492 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.733451    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bfd2407-f4a8-43c9-8994-58f0f6b254c4" (UID: "0bfd2407-f4a8-43c9-8994-58f0f6b254c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:46:42.833088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.833052    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:42.833088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.833080    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:42.833088 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.833089    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ktl\" (UniqueName: \"kubernetes.io/projected/0bfd2407-f4a8-43c9-8994-58f0f6b254c4-kube-api-access-68ktl\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:42.934137 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.934103    2560 generic.go:356] "Generic (PLEG): container finished" podID="0bfd2407-f4a8-43c9-8994-58f0f6b254c4" containerID="be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5" exitCode=0
Jul 07 15:46:42.934294 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.934149    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"
Jul 07 15:46:42.934294 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.934188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb" event={"ID":"0bfd2407-f4a8-43c9-8994-58f0f6b254c4","Type":"ContainerDied","Data":"be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5"}
Jul 07 15:46:42.934294 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.934227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb" event={"ID":"0bfd2407-f4a8-43c9-8994-58f0f6b254c4","Type":"ContainerDied","Data":"89600011f8a92d647c0bd44d93f7c4020cac5fb79946bd7ae114fdd6d3a337b3"}
Jul 07 15:46:42.934294 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.934244    2560 scope.go:117] "RemoveContainer" containerID="be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5"
Jul 07 15:46:42.943584 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.943565    2560 scope.go:117] "RemoveContainer" containerID="be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5"
Jul 07 15:46:42.943829 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:46:42.943811    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5\": container with ID starting with be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5 not found: ID does not exist" containerID="be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5"
Jul 07 15:46:42.943899 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.943837    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5"} err="failed to get container status \"be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5\": rpc error: code = NotFound desc = could not find container \"be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5\": container with ID starting with be4091559486409479abce2d81ec775ce1bb167f4b0d3f6e05bdc8c4c8606dc5 not found: ID does not exist"
Jul 07 15:46:42.958035 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.958011    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"]
Jul 07 15:46:42.962261 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:42.962240    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-99d0b0/test-model-99d0b0-kserve-8675cfc96f-n7hgb"]
Jul 07 15:46:43.138062 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.138027    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bfd2407-f4a8-43c9-8994-58f0f6b254c4" path="/var/lib/kubelet/pods/0bfd2407-f4a8-43c9-8994-58f0f6b254c4/volumes"
Jul 07 15:46:43.401828 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.401744    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb" podUID="edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" containerName="main" containerID="cri-o://0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe" gracePeriod=45
Jul 07 15:46:43.652714 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.652649    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:43.739532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739493    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tmp-dir\") pod \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") "
Jul 07 15:46:43.739532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739538    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-model-cache\") pod \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") "
Jul 07 15:46:43.739754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739564    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-home\") pod \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") "
Jul 07 15:46:43.739754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739598    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-dshm\") pod \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") "
Jul 07 15:46:43.739754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739628    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qznwf\" (UniqueName: \"kubernetes.io/projected/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-kube-api-access-qznwf\") pod \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") "
Jul 07 15:46:43.739754 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739652    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tls-certs\") pod \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\" (UID: \"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1\") "
Jul 07 15:46:43.739962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739876    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-home" (OuterVolumeSpecName: "home") pod "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" (UID: "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:43.739962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739870    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" (UID: "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:43.739962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.739894    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" (UID: "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:43.741932 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.741909    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-dshm" (OuterVolumeSpecName: "dshm") pod "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" (UID: "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:46:43.742017 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.741965    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" (UID: "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:46:43.742054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.742012    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-kube-api-access-qznwf" (OuterVolumeSpecName: "kube-api-access-qznwf") pod "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" (UID: "edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1"). InnerVolumeSpecName "kube-api-access-qznwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:46:43.840777 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.840738    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:43.840777 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.840771    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznwf\" (UniqueName: \"kubernetes.io/projected/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-kube-api-access-qznwf\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:43.840777 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.840781    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:43.840777 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.840790    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:43.841045 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.840798    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:43.841045 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.840805    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:46:43.941191 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.941104    2560 generic.go:356] "Generic (PLEG): container finished" podID="edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" containerID="0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe" exitCode=0
Jul 07 15:46:43.941191 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.941176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb" event={"ID":"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1","Type":"ContainerDied","Data":"0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe"}
Jul 07 15:46:43.941191 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.941183    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"
Jul 07 15:46:43.941460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.941207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb" event={"ID":"edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1","Type":"ContainerDied","Data":"5c5032ca85596fb2f1f2612f2c1eb0aeb2129f06050567cfdd248ede01dec3f0"}
Jul 07 15:46:43.941460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.941224    2560 scope.go:117] "RemoveContainer" containerID="0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe"
Jul 07 15:46:43.950105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.950083    2560 scope.go:117] "RemoveContainer" containerID="0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe"
Jul 07 15:46:43.950340 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:46:43.950318    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe\": container with ID starting with 0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe not found: ID does not exist" containerID="0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe"
Jul 07 15:46:43.950416 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.950347    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe"} err="failed to get container status \"0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe\": rpc error: code = NotFound desc = could not find container \"0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe\": container with ID starting with 0f3d6f074da183ff8338bcf859898ac79f6db0a78190d1c3883e50ce1a02befe not found: ID does not exist"
Jul 07 15:46:43.962397 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.962359    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"]
Jul 07 15:46:43.966605 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:43.966583    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ffde9f/test-model-ffde9f-kserve-79c8fdbff7-xxjgb"]
Jul 07 15:46:45.138256 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:46:45.138222    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" path="/var/lib/kubelet/pods/edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1/volumes"
Jul 07 15:47:12.588515 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.588473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"]
Jul 07 15:47:12.589054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.588717    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn" podUID="102ac0df-ffa2-415d-8bc5-be83f360b216" containerName="maas-api" containerID="cri-o://ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e" gracePeriod=30
Jul 07 15:47:12.841766 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.841704    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:47:12.897986 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.897957    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls\") pod \"102ac0df-ffa2-415d-8bc5-be83f360b216\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") "
Jul 07 15:47:12.898138 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.898005    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102ac0df-ffa2-415d-8bc5-be83f360b216-openshift-service-ca\") pod \"102ac0df-ffa2-415d-8bc5-be83f360b216\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") "
Jul 07 15:47:12.898138 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.898023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mfm6\" (UniqueName: \"kubernetes.io/projected/102ac0df-ffa2-415d-8bc5-be83f360b216-kube-api-access-9mfm6\") pod \"102ac0df-ffa2-415d-8bc5-be83f360b216\" (UID: \"102ac0df-ffa2-415d-8bc5-be83f360b216\") "
Jul 07 15:47:12.898366 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.898342    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102ac0df-ffa2-415d-8bc5-be83f360b216-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "102ac0df-ffa2-415d-8bc5-be83f360b216" (UID: "102ac0df-ffa2-415d-8bc5-be83f360b216"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:47:12.900093 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.900063    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "102ac0df-ffa2-415d-8bc5-be83f360b216" (UID: "102ac0df-ffa2-415d-8bc5-be83f360b216"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:47:12.900208 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.900121    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102ac0df-ffa2-415d-8bc5-be83f360b216-kube-api-access-9mfm6" (OuterVolumeSpecName: "kube-api-access-9mfm6") pod "102ac0df-ffa2-415d-8bc5-be83f360b216" (UID: "102ac0df-ffa2-415d-8bc5-be83f360b216"). InnerVolumeSpecName "kube-api-access-9mfm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:47:12.998865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.998831    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102ac0df-ffa2-415d-8bc5-be83f360b216-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:12.998865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.998860    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102ac0df-ffa2-415d-8bc5-be83f360b216-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:12.998865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:12.998871    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mfm6\" (UniqueName: \"kubernetes.io/projected/102ac0df-ffa2-415d-8bc5-be83f360b216-kube-api-access-9mfm6\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:13.070673 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.070638    2560 generic.go:356] "Generic (PLEG): container finished" podID="102ac0df-ffa2-415d-8bc5-be83f360b216" containerID="ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e" exitCode=0
Jul 07 15:47:13.070836 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.070685    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"
Jul 07 15:47:13.070836 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.070728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn" event={"ID":"102ac0df-ffa2-415d-8bc5-be83f360b216","Type":"ContainerDied","Data":"ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e"}
Jul 07 15:47:13.070836 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.070802    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn" event={"ID":"102ac0df-ffa2-415d-8bc5-be83f360b216","Type":"ContainerDied","Data":"a949807c5d923836d9c2a4e7c285cc6ca04f418cd04442cd5090228abcea0284"}
Jul 07 15:47:13.070836 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.070825    2560 scope.go:117] "RemoveContainer" containerID="ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e"
Jul 07 15:47:13.079306 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.079292    2560 scope.go:117] "RemoveContainer" containerID="ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e"
Jul 07 15:47:13.079586 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:47:13.079566    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e\": container with ID starting with ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e not found: ID does not exist" containerID="ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e"
Jul 07 15:47:13.079664 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.079597    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e"} err="failed to get container status \"ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e\": rpc error: code = NotFound desc = could not find container \"ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e\": container with ID starting with ed9a470fcce53d6ad1ff55960bc9eafd8580d26079684d4ac3976e79693fa77e not found: ID does not exist"
Jul 07 15:47:13.090831 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.090803    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"]
Jul 07 15:47:13.096346 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.096292    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-ffde9f-7b7d679689-qv8bn"]
Jul 07 15:47:13.137280 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:13.137253    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102ac0df-ffa2-415d-8bc5-be83f360b216" path="/var/lib/kubelet/pods/102ac0df-ffa2-415d-8bc5-be83f360b216/volumes"
Jul 07 15:47:22.793069 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793030    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"]
Jul 07 15:47:22.793620 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793600    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a71916-f2ef-40aa-adbb-f119fa568f4d" containerName="maas-api"
Jul 07 15:47:22.793723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793624    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a71916-f2ef-40aa-adbb-f119fa568f4d" containerName="maas-api"
Jul 07 15:47:22.793723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793639    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102ac0df-ffa2-415d-8bc5-be83f360b216" containerName="maas-api"
Jul 07 15:47:22.793723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793648    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="102ac0df-ffa2-415d-8bc5-be83f360b216" containerName="maas-api"
Jul 07 15:47:22.793723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793667    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bfd2407-f4a8-43c9-8994-58f0f6b254c4" containerName="main"
Jul 07 15:47:22.793723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793676    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bfd2407-f4a8-43c9-8994-58f0f6b254c4" containerName="main"
Jul 07 15:47:22.793723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793687    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" containerName="main"
Jul 07 15:47:22.793723 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793697    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" containerName="main"
Jul 07 15:47:22.793933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793782    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bfd2407-f4a8-43c9-8994-58f0f6b254c4" containerName="main"
Jul 07 15:47:22.793933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793795    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="102ac0df-ffa2-415d-8bc5-be83f360b216" containerName="maas-api"
Jul 07 15:47:22.793933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793805    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a71916-f2ef-40aa-adbb-f119fa568f4d" containerName="maas-api"
Jul 07 15:47:22.793933 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.793815    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcaa7c3-2a97-40d3-abf3-dc5701d6d1c1" containerName="main"
Jul 07 15:47:22.798447 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.798426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.801124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.801102    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-face20-serving-cert\""
Jul 07 15:47:22.815436 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.815364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"]
Jul 07 15:47:22.882975 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.882931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.883170 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.883015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2mq\" (UniqueName: \"kubernetes.io/projected/fc13704f-b47d-429d-b43b-c7db5a05ea50-kube-api-access-7h2mq\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.883170 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.883055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13704f-b47d-429d-b43b-c7db5a05ea50-openshift-service-ca\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.983755 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.983723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13704f-b47d-429d-b43b-c7db5a05ea50-openshift-service-ca\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.983949 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.983783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.983949 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.983831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2mq\" (UniqueName: \"kubernetes.io/projected/fc13704f-b47d-429d-b43b-c7db5a05ea50-kube-api-access-7h2mq\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.984069 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:47:22.983955    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-face20-serving-cert: secret "maas-api-e2e-api-a-face20-serving-cert" not found
Jul 07 15:47:22.984069 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:47:22.984033    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls podName:fc13704f-b47d-429d-b43b-c7db5a05ea50 nodeName:}" failed. No retries permitted until 2026-07-07 15:47:23.484011393 +0000 UTC m=+2060.892120290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls") pod "maas-api-e2e-api-a-face20-6cbc44fd98-94mxh" (UID: "fc13704f-b47d-429d-b43b-c7db5a05ea50") : secret "maas-api-e2e-api-a-face20-serving-cert" not found
Jul 07 15:47:22.984566 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.984543    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13704f-b47d-429d-b43b-c7db5a05ea50-openshift-service-ca\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:22.997406 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:22.997376    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2mq\" (UniqueName: \"kubernetes.io/projected/fc13704f-b47d-429d-b43b-c7db5a05ea50-kube-api-access-7h2mq\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:23.487813 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:23.487782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:23.490422 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:23.490374    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls\") pod \"maas-api-e2e-api-a-face20-6cbc44fd98-94mxh\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") " pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:23.709142 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:23.709104    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:23.850149 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:23.850125    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"]
Jul 07 15:47:23.852367 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:47:23.852338    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc13704f_b47d_429d_b43b_c7db5a05ea50.slice/crio-6cac032456a5cb547428d150df967383288cb85784b865db1545fa3ba1630f08 WatchSource:0}: Error finding container 6cac032456a5cb547428d150df967383288cb85784b865db1545fa3ba1630f08: Status 404 returned error can't find the container with id 6cac032456a5cb547428d150df967383288cb85784b865db1545fa3ba1630f08
Jul 07 15:47:24.124659 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:24.124570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh" event={"ID":"fc13704f-b47d-429d-b43b-c7db5a05ea50","Type":"ContainerStarted","Data":"6cac032456a5cb547428d150df967383288cb85784b865db1545fa3ba1630f08"}
Jul 07 15:47:25.135667 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:25.135637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh" event={"ID":"fc13704f-b47d-429d-b43b-c7db5a05ea50","Type":"ContainerStarted","Data":"ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737"}
Jul 07 15:47:25.136045 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:25.135692    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:25.149483 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:25.149434    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh" podStartSLOduration=2.585170185 podStartE2EDuration="3.149418104s" totalImagesPullingTime="564.247919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:25.148054925 +0000 UTC m=+2062.556163830" watchObservedRunningTime="2026-07-07 15:47:25.149418104 +0000 UTC m=+2062.557527006"
Jul 07 15:47:30.797970 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.797925    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"]
Jul 07 15:47:30.802005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.801980    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.804494 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.804470    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-7c4422-serving-cert\""
Jul 07 15:47:30.811643 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.811621    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"]
Jul 07 15:47:30.859622 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.859587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b32538-7cb1-4699-b720-1af98c250cae-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.859802 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.859673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.859802 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.859698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dl5\" (UniqueName: \"kubernetes.io/projected/a4b32538-7cb1-4699-b720-1af98c250cae-kube-api-access-n4dl5\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.960321 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.960283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.960321 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.960327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dl5\" (UniqueName: \"kubernetes.io/projected/a4b32538-7cb1-4699-b720-1af98c250cae-kube-api-access-n4dl5\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.960608 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.960370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b32538-7cb1-4699-b720-1af98c250cae-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.960608 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:47:30.960499    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-7c4422-serving-cert: secret "maas-api-e2e-api-b-7c4422-serving-cert" not found
Jul 07 15:47:30.960608 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:47:30.960581    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls podName:a4b32538-7cb1-4699-b720-1af98c250cae nodeName:}" failed. No retries permitted until 2026-07-07 15:47:31.460559755 +0000 UTC m=+2068.868668649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls") pod "maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5" (UID: "a4b32538-7cb1-4699-b720-1af98c250cae") : secret "maas-api-e2e-api-b-7c4422-serving-cert" not found
Jul 07 15:47:30.961014 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.960996    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b32538-7cb1-4699-b720-1af98c250cae-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:30.968800 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:30.968772    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dl5\" (UniqueName: \"kubernetes.io/projected/a4b32538-7cb1-4699-b720-1af98c250cae-kube-api-access-n4dl5\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:31.140683 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:31.140657    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:31.464698 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:31.464603    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:31.467005 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:31.466988    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls\") pod \"maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") " pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:31.716061 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:31.715973    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:31.855330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:31.855151    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"]
Jul 07 15:47:31.858265 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:47:31.858239    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b32538_7cb1_4699_b720_1af98c250cae.slice/crio-17b2b48e340a8229b66e1c5ebe73e3782105fbcdbbb25679368bfacb6625e325 WatchSource:0}: Error finding container 17b2b48e340a8229b66e1c5ebe73e3782105fbcdbbb25679368bfacb6625e325: Status 404 returned error can't find the container with id 17b2b48e340a8229b66e1c5ebe73e3782105fbcdbbb25679368bfacb6625e325
Jul 07 15:47:32.162478 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:32.162442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5" event={"ID":"a4b32538-7cb1-4699-b720-1af98c250cae","Type":"ContainerStarted","Data":"17b2b48e340a8229b66e1c5ebe73e3782105fbcdbbb25679368bfacb6625e325"}
Jul 07 15:47:33.168620 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:33.168581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5" event={"ID":"a4b32538-7cb1-4699-b720-1af98c250cae","Type":"ContainerStarted","Data":"79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91"}
Jul 07 15:47:33.169038 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:33.168696    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:33.187748 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:33.187700    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5" podStartSLOduration=2.81766324 podStartE2EDuration="3.187681435s" totalImagesPullingTime="370.018195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:33.185970956 +0000 UTC m=+2070.594079860" watchObservedRunningTime="2026-07-07 15:47:33.187681435 +0000 UTC m=+2070.595790338"
Jul 07 15:47:39.177887 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:39.177859    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:43.236427 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.236379    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"]
Jul 07 15:47:43.236944 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.236599    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh" podUID="fc13704f-b47d-429d-b43b-c7db5a05ea50" containerName="maas-api" containerID="cri-o://ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737" gracePeriod=30
Jul 07 15:47:43.495166 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.495102    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:43.578837 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.578805    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13704f-b47d-429d-b43b-c7db5a05ea50-openshift-service-ca\") pod \"fc13704f-b47d-429d-b43b-c7db5a05ea50\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") "
Jul 07 15:47:43.579033 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.578898    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h2mq\" (UniqueName: \"kubernetes.io/projected/fc13704f-b47d-429d-b43b-c7db5a05ea50-kube-api-access-7h2mq\") pod \"fc13704f-b47d-429d-b43b-c7db5a05ea50\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") "
Jul 07 15:47:43.579102 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.579052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls\") pod \"fc13704f-b47d-429d-b43b-c7db5a05ea50\" (UID: \"fc13704f-b47d-429d-b43b-c7db5a05ea50\") "
Jul 07 15:47:43.579253 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.579225    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc13704f-b47d-429d-b43b-c7db5a05ea50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc13704f-b47d-429d-b43b-c7db5a05ea50" (UID: "fc13704f-b47d-429d-b43b-c7db5a05ea50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:47:43.579435 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.579379    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc13704f-b47d-429d-b43b-c7db5a05ea50-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:43.581044 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.581024    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc13704f-b47d-429d-b43b-c7db5a05ea50" (UID: "fc13704f-b47d-429d-b43b-c7db5a05ea50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:47:43.581111 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.581090    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc13704f-b47d-429d-b43b-c7db5a05ea50-kube-api-access-7h2mq" (OuterVolumeSpecName: "kube-api-access-7h2mq") pod "fc13704f-b47d-429d-b43b-c7db5a05ea50" (UID: "fc13704f-b47d-429d-b43b-c7db5a05ea50"). InnerVolumeSpecName "kube-api-access-7h2mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:47:43.680949 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.680912    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2mq\" (UniqueName: \"kubernetes.io/projected/fc13704f-b47d-429d-b43b-c7db5a05ea50-kube-api-access-7h2mq\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:43.680949 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:43.680942    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc13704f-b47d-429d-b43b-c7db5a05ea50-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:44.218981 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.218939    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc13704f-b47d-429d-b43b-c7db5a05ea50" containerID="ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737" exitCode=0
Jul 07 15:47:44.218981 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.218985    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"
Jul 07 15:47:44.219208 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.219007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh" event={"ID":"fc13704f-b47d-429d-b43b-c7db5a05ea50","Type":"ContainerDied","Data":"ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737"}
Jul 07 15:47:44.219208 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.219058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh" event={"ID":"fc13704f-b47d-429d-b43b-c7db5a05ea50","Type":"ContainerDied","Data":"6cac032456a5cb547428d150df967383288cb85784b865db1545fa3ba1630f08"}
Jul 07 15:47:44.219208 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.219079    2560 scope.go:117] "RemoveContainer" containerID="ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737"
Jul 07 15:47:44.228015 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.227995    2560 scope.go:117] "RemoveContainer" containerID="ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737"
Jul 07 15:47:44.228280 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:47:44.228262    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737\": container with ID starting with ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737 not found: ID does not exist" containerID="ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737"
Jul 07 15:47:44.228326 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.228288    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737"} err="failed to get container status \"ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737\": rpc error: code = NotFound desc = could not find container \"ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737\": container with ID starting with ed11a865ebfa8b1edbcab40deff754cc562db052eb03c3220c878448e2319737 not found: ID does not exist"
Jul 07 15:47:44.244031 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.244006    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"]
Jul 07 15:47:44.247576 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:44.247550    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-face20-6cbc44fd98-94mxh"]
Jul 07 15:47:45.137611 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:45.137573    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc13704f-b47d-429d-b43b-c7db5a05ea50" path="/var/lib/kubelet/pods/fc13704f-b47d-429d-b43b-c7db5a05ea50/volumes"
Jul 07 15:47:51.252300 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.252222    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"]
Jul 07 15:47:51.252908 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.252524    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5" podUID="a4b32538-7cb1-4699-b720-1af98c250cae" containerName="maas-api" containerID="cri-o://79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91" gracePeriod=30
Jul 07 15:47:51.504998 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.504936    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:51.559767 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.559740    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4dl5\" (UniqueName: \"kubernetes.io/projected/a4b32538-7cb1-4699-b720-1af98c250cae-kube-api-access-n4dl5\") pod \"a4b32538-7cb1-4699-b720-1af98c250cae\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") "
Jul 07 15:47:51.559935 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.559772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls\") pod \"a4b32538-7cb1-4699-b720-1af98c250cae\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") "
Jul 07 15:47:51.559935 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.559801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b32538-7cb1-4699-b720-1af98c250cae-openshift-service-ca\") pod \"a4b32538-7cb1-4699-b720-1af98c250cae\" (UID: \"a4b32538-7cb1-4699-b720-1af98c250cae\") "
Jul 07 15:47:51.560192 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.560156    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b32538-7cb1-4699-b720-1af98c250cae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4b32538-7cb1-4699-b720-1af98c250cae" (UID: "a4b32538-7cb1-4699-b720-1af98c250cae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:47:51.561839 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.561813    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4b32538-7cb1-4699-b720-1af98c250cae" (UID: "a4b32538-7cb1-4699-b720-1af98c250cae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:47:51.561956 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.561938    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b32538-7cb1-4699-b720-1af98c250cae-kube-api-access-n4dl5" (OuterVolumeSpecName: "kube-api-access-n4dl5") pod "a4b32538-7cb1-4699-b720-1af98c250cae" (UID: "a4b32538-7cb1-4699-b720-1af98c250cae"). InnerVolumeSpecName "kube-api-access-n4dl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:47:51.660596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.660547    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dl5\" (UniqueName: \"kubernetes.io/projected/a4b32538-7cb1-4699-b720-1af98c250cae-kube-api-access-n4dl5\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:51.660596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.660590    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b32538-7cb1-4699-b720-1af98c250cae-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:51.660596 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:51.660601    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b32538-7cb1-4699-b720-1af98c250cae-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:47:52.260890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.260851    2560 generic.go:356] "Generic (PLEG): container finished" podID="a4b32538-7cb1-4699-b720-1af98c250cae" containerID="79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91" exitCode=0
Jul 07 15:47:52.260890 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.260892    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"
Jul 07 15:47:52.261356 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.260923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5" event={"ID":"a4b32538-7cb1-4699-b720-1af98c250cae","Type":"ContainerDied","Data":"79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91"}
Jul 07 15:47:52.261356 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.260973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5" event={"ID":"a4b32538-7cb1-4699-b720-1af98c250cae","Type":"ContainerDied","Data":"17b2b48e340a8229b66e1c5ebe73e3782105fbcdbbb25679368bfacb6625e325"}
Jul 07 15:47:52.261356 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.260997    2560 scope.go:117] "RemoveContainer" containerID="79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91"
Jul 07 15:47:52.270012 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.269993    2560 scope.go:117] "RemoveContainer" containerID="79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91"
Jul 07 15:47:52.270309 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:47:52.270292    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91\": container with ID starting with 79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91 not found: ID does not exist" containerID="79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91"
Jul 07 15:47:52.270372 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.270321    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91"} err="failed to get container status \"79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91\": rpc error: code = NotFound desc = could not find container \"79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91\": container with ID starting with 79deeb5240c85dcc6175371ba064d93c2ed5c5ed566138f6c178bcf4abc85a91 not found: ID does not exist"
Jul 07 15:47:52.282334 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.282306    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"]
Jul 07 15:47:52.286366 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:52.286345    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-7c4422-7559f4f87b-fqfk5"]
Jul 07 15:47:53.136796 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:47:53.136762    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b32538-7cb1-4699-b720-1af98c250cae" path="/var/lib/kubelet/pods/a4b32538-7cb1-4699-b720-1af98c250cae/volumes"
Jul 07 15:48:01.049241 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.049203    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"]
Jul 07 15:48:01.049768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.049651    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc13704f-b47d-429d-b43b-c7db5a05ea50" containerName="maas-api"
Jul 07 15:48:01.049768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.049669    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc13704f-b47d-429d-b43b-c7db5a05ea50" containerName="maas-api"
Jul 07 15:48:01.049768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.049688    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b32538-7cb1-4699-b720-1af98c250cae" containerName="maas-api"
Jul 07 15:48:01.049768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.049693    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b32538-7cb1-4699-b720-1af98c250cae" containerName="maas-api"
Jul 07 15:48:01.049768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.049755    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc13704f-b47d-429d-b43b-c7db5a05ea50" containerName="maas-api"
Jul 07 15:48:01.049768 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.049766    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b32538-7cb1-4699-b720-1af98c250cae" containerName="maas-api"
Jul 07 15:48:01.053010 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.052988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.055736 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.055710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-1f2ee2-serving-cert\""
Jul 07 15:48:01.063250 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.063224    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"]
Jul 07 15:48:01.150337 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.150303    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.150533 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.150352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnzzt\" (UniqueName: \"kubernetes.io/projected/b1852bed-917c-469d-8c16-813af31f5079-kube-api-access-xnzzt\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.150533 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.150473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1852bed-917c-469d-8c16-813af31f5079-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.251897 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.251828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1852bed-917c-469d-8c16-813af31f5079-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.252082 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.251956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.252082 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.252009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnzzt\" (UniqueName: \"kubernetes.io/projected/b1852bed-917c-469d-8c16-813af31f5079-kube-api-access-xnzzt\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.252157 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:48:01.252122    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-1f2ee2-serving-cert: secret "maas-api-e2e-shared-a-1f2ee2-serving-cert" not found
Jul 07 15:48:01.252228 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:48:01.252213    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls podName:b1852bed-917c-469d-8c16-813af31f5079 nodeName:}" failed. No retries permitted until 2026-07-07 15:48:01.752189645 +0000 UTC m=+2099.160298526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls") pod "maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66" (UID: "b1852bed-917c-469d-8c16-813af31f5079") : secret "maas-api-e2e-shared-a-1f2ee2-serving-cert" not found
Jul 07 15:48:01.252653 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.252635    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1852bed-917c-469d-8c16-813af31f5079-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.262961 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.262934    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnzzt\" (UniqueName: \"kubernetes.io/projected/b1852bed-917c-469d-8c16-813af31f5079-kube-api-access-xnzzt\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.756382 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.756341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.758959 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.758927    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls\") pod \"maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") " pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:01.964330 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:01.964297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:02.118771 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:02.118744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"]
Jul 07 15:48:02.120681 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:48:02.120656    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1852bed_917c_469d_8c16_813af31f5079.slice/crio-98a921242313ad85fd467dc1c582a555301ae3f2d3817d1fb64ea16bd1c11878 WatchSource:0}: Error finding container 98a921242313ad85fd467dc1c582a555301ae3f2d3817d1fb64ea16bd1c11878: Status 404 returned error can't find the container with id 98a921242313ad85fd467dc1c582a555301ae3f2d3817d1fb64ea16bd1c11878
Jul 07 15:48:02.307847 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:02.307759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66" event={"ID":"b1852bed-917c-469d-8c16-813af31f5079","Type":"ContainerStarted","Data":"98a921242313ad85fd467dc1c582a555301ae3f2d3817d1fb64ea16bd1c11878"}
Jul 07 15:48:03.319482 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:03.319437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66" event={"ID":"b1852bed-917c-469d-8c16-813af31f5079","Type":"ContainerStarted","Data":"d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705"}
Jul 07 15:48:03.319891 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:03.319582    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:03.337851 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:03.337796    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66" podStartSLOduration=1.992593544 podStartE2EDuration="2.33778169s" totalImagesPullingTime="345.188146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:03.335912173 +0000 UTC m=+2100.744021074" watchObservedRunningTime="2026-07-07 15:48:03.33778169 +0000 UTC m=+2100.745890592"
Jul 07 15:48:09.057516 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.057478    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"]
Jul 07 15:48:09.061108 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.061089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.063542 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.063520    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-458f34-serving-cert\""
Jul 07 15:48:09.074299 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.074266    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"]
Jul 07 15:48:09.122451 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.122404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6z2\" (UniqueName: \"kubernetes.io/projected/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-kube-api-access-xp6z2\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.122451 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.122457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.122681 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.122555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.223435 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.223370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.223613 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.223469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6z2\" (UniqueName: \"kubernetes.io/projected/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-kube-api-access-xp6z2\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.223613 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.223505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.223613 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:48:09.223508    2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-458f34-serving-cert: secret "maas-api-e2e-shared-b-458f34-serving-cert" not found
Jul 07 15:48:09.223613 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:48:09.223567    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls podName:55368b3e-2ec0-40ce-836d-5a4f37e06bf7 nodeName:}" failed. No retries permitted until 2026-07-07 15:48:09.723551177 +0000 UTC m=+2107.131660060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls") pod "maas-api-e2e-shared-b-458f34-8c54f49-q6hg7" (UID: "55368b3e-2ec0-40ce-836d-5a4f37e06bf7") : secret "maas-api-e2e-shared-b-458f34-serving-cert" not found
Jul 07 15:48:09.224126 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.224108    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.231726 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.231700    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6z2\" (UniqueName: \"kubernetes.io/projected/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-kube-api-access-xp6z2\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.329657 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.329576    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:48:09.730094 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.730055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.732546 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.732521    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls\") pod \"maas-api-e2e-shared-b-458f34-8c54f49-q6hg7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") " pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:09.983855 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:09.983766    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:10.122264 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:10.122137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"]
Jul 07 15:48:10.124433 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:48:10.124406    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55368b3e_2ec0_40ce_836d_5a4f37e06bf7.slice/crio-f1867b4f8852a57695c35e58659be2ae563297714a9469679000f864fcd13c3d WatchSource:0}: Error finding container f1867b4f8852a57695c35e58659be2ae563297714a9469679000f864fcd13c3d: Status 404 returned error can't find the container with id f1867b4f8852a57695c35e58659be2ae563297714a9469679000f864fcd13c3d
Jul 07 15:48:10.351218 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:10.351131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7" event={"ID":"55368b3e-2ec0-40ce-836d-5a4f37e06bf7","Type":"ContainerStarted","Data":"f1867b4f8852a57695c35e58659be2ae563297714a9469679000f864fcd13c3d"}
Jul 07 15:48:11.356616 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:11.356582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7" event={"ID":"55368b3e-2ec0-40ce-836d-5a4f37e06bf7","Type":"ContainerStarted","Data":"44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a"}
Jul 07 15:48:11.357008 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:11.356702    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:11.374014 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:11.373966    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7" podStartSLOduration=1.764930803 podStartE2EDuration="2.373949203s" totalImagesPullingTime="609.0184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:11.371759005 +0000 UTC m=+2108.779867906" watchObservedRunningTime="2026-07-07 15:48:11.373949203 +0000 UTC m=+2108.782058105"
Jul 07 15:48:17.367266 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:17.367236    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:48:21.435591 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.435559    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"]
Jul 07 15:48:21.438942 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.438925    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.441311 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.441287    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"openshift-service-ca.crt\""
Jul 07 15:48:21.441311 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.441300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"default-dockercfg-7g94j\""
Jul 07 15:48:21.442222 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.442200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"auth-test-model-1f2ee2-kserve-self-signed-certs\""
Jul 07 15:48:21.442323 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.442238    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"kube-root-ca.crt\""
Jul 07 15:48:21.451370 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.449467    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"]
Jul 07 15:48:21.536629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.536588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-model-cache\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.536629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.536628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fede4a-7415-4414-ac2c-f0e8584067c4-tls-certs\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.536860 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.536721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qmh7\" (UniqueName: \"kubernetes.io/projected/78fede4a-7415-4414-ac2c-f0e8584067c4-kube-api-access-7qmh7\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.536860 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.536759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-dshm\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.536860 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.536784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-home\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.536860 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.536826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-tmp-dir\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638237 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-model-cache\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638237 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fede4a-7415-4414-ac2c-f0e8584067c4-tls-certs\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638521 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qmh7\" (UniqueName: \"kubernetes.io/projected/78fede4a-7415-4414-ac2c-f0e8584067c4-kube-api-access-7qmh7\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638521 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-dshm\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638521 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-home\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638521 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-tmp-dir\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638746 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638719    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-model-cache\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638797 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638763    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-home\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.638889 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.638873    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-tmp-dir\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.640719 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.640691    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-dshm\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.641014 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.640992    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fede4a-7415-4414-ac2c-f0e8584067c4-tls-certs\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.646655 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.646632    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qmh7\" (UniqueName: \"kubernetes.io/projected/78fede4a-7415-4414-ac2c-f0e8584067c4-kube-api-access-7qmh7\") pod \"auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") " pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.754815 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.754718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:21.887720 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:21.887693    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"]
Jul 07 15:48:21.889772 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:48:21.889739    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78fede4a_7415_4414_ac2c_f0e8584067c4.slice/crio-4a341889bd7a7b8e69d34bf1f4241aad3b69e63378145e6d8f01f73321cb224b WatchSource:0}: Error finding container 4a341889bd7a7b8e69d34bf1f4241aad3b69e63378145e6d8f01f73321cb224b: Status 404 returned error can't find the container with id 4a341889bd7a7b8e69d34bf1f4241aad3b69e63378145e6d8f01f73321cb224b
Jul 07 15:48:22.405550 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:22.405514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5" event={"ID":"78fede4a-7415-4414-ac2c-f0e8584067c4","Type":"ContainerStarted","Data":"e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc"}
Jul 07 15:48:22.405550 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:22.405554    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5" event={"ID":"78fede4a-7415-4414-ac2c-f0e8584067c4","Type":"ContainerStarted","Data":"4a341889bd7a7b8e69d34bf1f4241aad3b69e63378145e6d8f01f73321cb224b"}
Jul 07 15:48:22.421907 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:22.421847    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5" podStartSLOduration=1.421828232 podStartE2EDuration="1.421828232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:22.420835411 +0000 UTC m=+2119.828944314" watchObservedRunningTime="2026-07-07 15:48:22.421828232 +0000 UTC m=+2119.829937134"
Jul 07 15:48:31.755465 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:31.755420    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:31.755465 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:31.755474    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:31.768302 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:31.768276    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:32.461489 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:32.461459    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:48:54.548694 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.548484    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"]
Jul 07 15:48:54.554494 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.554459    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.557274 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.557247    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"openshift-service-ca.crt\""
Jul 07 15:48:54.558269 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.558243    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"default-dockercfg-n5tf6\""
Jul 07 15:48:54.558409 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.558244    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"auth-test-model-458f34-kserve-self-signed-certs\""
Jul 07 15:48:54.558409 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.558259    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"kube-root-ca.crt\""
Jul 07 15:48:54.560468 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.560447    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"]
Jul 07 15:48:54.636372 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.636331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-home\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.636372 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.636374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-model-cache\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.636629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.636425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-tmp-dir\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.636629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.636506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6840207c-050c-432a-b36b-2becf3861d67-tls-certs\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.636629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.636580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-dshm\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.636752 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.636626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp95m\" (UniqueName: \"kubernetes.io/projected/6840207c-050c-432a-b36b-2becf3861d67-kube-api-access-gp95m\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.737470 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.737422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-dshm\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.737667 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.737479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp95m\" (UniqueName: \"kubernetes.io/projected/6840207c-050c-432a-b36b-2becf3861d67-kube-api-access-gp95m\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.737667 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.737569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-home\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.737667 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.737590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-model-cache\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.737667 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.737608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-tmp-dir\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.737848 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.737758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6840207c-050c-432a-b36b-2becf3861d67-tls-certs\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.738015 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.737986    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-tmp-dir\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.738108 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.738012    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-home\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.738108 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.738051    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-model-cache\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.739749 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.739720    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-dshm\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.740289 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.740265    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6840207c-050c-432a-b36b-2becf3861d67-tls-certs\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.748457 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.748431    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp95m\" (UniqueName: \"kubernetes.io/projected/6840207c-050c-432a-b36b-2becf3861d67-kube-api-access-gp95m\") pod \"auth-test-model-458f34-kserve-7bfbfbc467-nzrm2\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") " pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:54.866208 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:54.866114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:48:55.003439 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:55.003359    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"]
Jul 07 15:48:55.005604 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:48:55.005566    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6840207c_050c_432a_b36b_2becf3861d67.slice/crio-2af98687565edf8962e0f27a0998d3e260223e02a91d6e0bbbc4920ddfe2586d WatchSource:0}: Error finding container 2af98687565edf8962e0f27a0998d3e260223e02a91d6e0bbbc4920ddfe2586d: Status 404 returned error can't find the container with id 2af98687565edf8962e0f27a0998d3e260223e02a91d6e0bbbc4920ddfe2586d
Jul 07 15:48:55.562896 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:55.562859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2" event={"ID":"6840207c-050c-432a-b36b-2becf3861d67","Type":"ContainerStarted","Data":"610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783"}
Jul 07 15:48:55.562896 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:55.562901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2" event={"ID":"6840207c-050c-432a-b36b-2becf3861d67","Type":"ContainerStarted","Data":"2af98687565edf8962e0f27a0998d3e260223e02a91d6e0bbbc4920ddfe2586d"}
Jul 07 15:48:55.581957 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:48:55.581897    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2" podStartSLOduration=1.581880357 podStartE2EDuration="1.581880357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:55.580320544 +0000 UTC m=+2152.988429450" watchObservedRunningTime="2026-07-07 15:48:55.581880357 +0000 UTC m=+2152.989989275"
Jul 07 15:49:04.866513 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:49:04.866470    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:49:04.866513 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:49:04.866524    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:49:04.879083 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:49:04.879053    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:49:05.623141 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:49:05.623110    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:50:51.797988 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:50:51.797904    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"]
Jul 07 15:50:52.674594 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:50:52.674557    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"]
Jul 07 15:51:06.818174 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:06.818127    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5" podUID="78fede4a-7415-4414-ac2c-f0e8584067c4" containerName="main" containerID="cri-o://e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc" gracePeriod=45
Jul 07 15:51:07.068221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.068158    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:51:07.175460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.175421    2560 generic.go:356] "Generic (PLEG): container finished" podID="78fede4a-7415-4414-ac2c-f0e8584067c4" containerID="e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc" exitCode=0
Jul 07 15:51:07.175460 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.175462    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"
Jul 07 15:51:07.175671 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.175486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5" event={"ID":"78fede4a-7415-4414-ac2c-f0e8584067c4","Type":"ContainerDied","Data":"e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc"}
Jul 07 15:51:07.175671 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.175529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5" event={"ID":"78fede4a-7415-4414-ac2c-f0e8584067c4","Type":"ContainerDied","Data":"4a341889bd7a7b8e69d34bf1f4241aad3b69e63378145e6d8f01f73321cb224b"}
Jul 07 15:51:07.175671 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.175547    2560 scope.go:117] "RemoveContainer" containerID="e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc"
Jul 07 15:51:07.175930 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.175915    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fede4a-7415-4414-ac2c-f0e8584067c4-tls-certs\") pod \"78fede4a-7415-4414-ac2c-f0e8584067c4\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") "
Jul 07 15:51:07.175992 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.175979    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-model-cache\") pod \"78fede4a-7415-4414-ac2c-f0e8584067c4\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") "
Jul 07 15:51:07.176063 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-tmp-dir\") pod \"78fede4a-7415-4414-ac2c-f0e8584067c4\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") "
Jul 07 15:51:07.176211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-home\") pod \"78fede4a-7415-4414-ac2c-f0e8584067c4\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") "
Jul 07 15:51:07.176211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176133    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qmh7\" (UniqueName: \"kubernetes.io/projected/78fede4a-7415-4414-ac2c-f0e8584067c4-kube-api-access-7qmh7\") pod \"78fede4a-7415-4414-ac2c-f0e8584067c4\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") "
Jul 07 15:51:07.176211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-dshm\") pod \"78fede4a-7415-4414-ac2c-f0e8584067c4\" (UID: \"78fede4a-7415-4414-ac2c-f0e8584067c4\") "
Jul 07 15:51:07.176408 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176258    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78fede4a-7415-4414-ac2c-f0e8584067c4" (UID: "78fede4a-7415-4414-ac2c-f0e8584067c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:07.176408 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176364    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-home" (OuterVolumeSpecName: "home") pod "78fede4a-7415-4414-ac2c-f0e8584067c4" (UID: "78fede4a-7415-4414-ac2c-f0e8584067c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:07.176527 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176447    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "78fede4a-7415-4414-ac2c-f0e8584067c4" (UID: "78fede4a-7415-4414-ac2c-f0e8584067c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:07.176591 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176538    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:07.176591 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176563    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:07.176591 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.176579    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:07.178571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.178545    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-dshm" (OuterVolumeSpecName: "dshm") pod "78fede4a-7415-4414-ac2c-f0e8584067c4" (UID: "78fede4a-7415-4414-ac2c-f0e8584067c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:07.178571 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.178559    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fede4a-7415-4414-ac2c-f0e8584067c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78fede4a-7415-4414-ac2c-f0e8584067c4" (UID: "78fede4a-7415-4414-ac2c-f0e8584067c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:51:07.178712 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.178565    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fede4a-7415-4414-ac2c-f0e8584067c4-kube-api-access-7qmh7" (OuterVolumeSpecName: "kube-api-access-7qmh7") pod "78fede4a-7415-4414-ac2c-f0e8584067c4" (UID: "78fede4a-7415-4414-ac2c-f0e8584067c4"). InnerVolumeSpecName "kube-api-access-7qmh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:51:07.195383 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.195363    2560 scope.go:117] "RemoveContainer" containerID="e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc"
Jul 07 15:51:07.195715 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:51:07.195692    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc\": container with ID starting with e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc not found: ID does not exist" containerID="e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc"
Jul 07 15:51:07.195764 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.195725    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc"} err="failed to get container status \"e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc\": rpc error: code = NotFound desc = could not find container \"e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc\": container with ID starting with e473a991d818a18fddf1b60abac652db94e768e6211476f111698a0c53a654fc not found: ID does not exist"
Jul 07 15:51:07.277465 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.277433    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fede4a-7415-4414-ac2c-f0e8584067c4-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:07.277465 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.277462    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fede4a-7415-4414-ac2c-f0e8584067c4-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:07.277465 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.277472    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qmh7\" (UniqueName: \"kubernetes.io/projected/78fede4a-7415-4414-ac2c-f0e8584067c4-kube-api-access-7qmh7\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:07.500903 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.500873    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"]
Jul 07 15:51:07.506771 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.506744    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/auth-test-model-1f2ee2-kserve-64fcd7846c-ddtr5"]
Jul 07 15:51:07.694889 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.694816    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2" podUID="6840207c-050c-432a-b36b-2becf3861d67" containerName="main" containerID="cri-o://610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783" gracePeriod=45
Jul 07 15:51:07.943223 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:07.943198    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:51:08.084710 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.084612    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-dshm\") pod \"6840207c-050c-432a-b36b-2becf3861d67\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") "
Jul 07 15:51:08.084710 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.084667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6840207c-050c-432a-b36b-2becf3861d67-tls-certs\") pod \"6840207c-050c-432a-b36b-2becf3861d67\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") "
Jul 07 15:51:08.084937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.084722    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-home\") pod \"6840207c-050c-432a-b36b-2becf3861d67\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") "
Jul 07 15:51:08.084937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.084777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-model-cache\") pod \"6840207c-050c-432a-b36b-2becf3861d67\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") "
Jul 07 15:51:08.084937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.084817    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-tmp-dir\") pod \"6840207c-050c-432a-b36b-2becf3861d67\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") "
Jul 07 15:51:08.084937 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.084847    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp95m\" (UniqueName: \"kubernetes.io/projected/6840207c-050c-432a-b36b-2becf3861d67-kube-api-access-gp95m\") pod \"6840207c-050c-432a-b36b-2becf3861d67\" (UID: \"6840207c-050c-432a-b36b-2becf3861d67\") "
Jul 07 15:51:08.085144 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.085084    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6840207c-050c-432a-b36b-2becf3861d67" (UID: "6840207c-050c-432a-b36b-2becf3861d67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:08.085144 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.085091    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-home" (OuterVolumeSpecName: "home") pod "6840207c-050c-432a-b36b-2becf3861d67" (UID: "6840207c-050c-432a-b36b-2becf3861d67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:08.085144 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.085111    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-model-cache" (OuterVolumeSpecName: "model-cache") pod "6840207c-050c-432a-b36b-2becf3861d67" (UID: "6840207c-050c-432a-b36b-2becf3861d67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:08.085283 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.085205    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:08.085283 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.085225    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:08.085283 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.085243    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:08.086943 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.086920    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-dshm" (OuterVolumeSpecName: "dshm") pod "6840207c-050c-432a-b36b-2becf3861d67" (UID: "6840207c-050c-432a-b36b-2becf3861d67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:51:08.087054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.086999    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6840207c-050c-432a-b36b-2becf3861d67-kube-api-access-gp95m" (OuterVolumeSpecName: "kube-api-access-gp95m") pod "6840207c-050c-432a-b36b-2becf3861d67" (UID: "6840207c-050c-432a-b36b-2becf3861d67"). InnerVolumeSpecName "kube-api-access-gp95m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:51:08.087054 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.087043    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6840207c-050c-432a-b36b-2becf3861d67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6840207c-050c-432a-b36b-2becf3861d67" (UID: "6840207c-050c-432a-b36b-2becf3861d67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:51:08.183002 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.182961    2560 generic.go:356] "Generic (PLEG): container finished" podID="6840207c-050c-432a-b36b-2becf3861d67" containerID="610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783" exitCode=0
Jul 07 15:51:08.183181 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.183019    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"
Jul 07 15:51:08.183181 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.183031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2" event={"ID":"6840207c-050c-432a-b36b-2becf3861d67","Type":"ContainerDied","Data":"610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783"}
Jul 07 15:51:08.183181 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.183067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2" event={"ID":"6840207c-050c-432a-b36b-2becf3861d67","Type":"ContainerDied","Data":"2af98687565edf8962e0f27a0998d3e260223e02a91d6e0bbbc4920ddfe2586d"}
Jul 07 15:51:08.183181 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.183087    2560 scope.go:117] "RemoveContainer" containerID="610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783"
Jul 07 15:51:08.185859 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.185837    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp95m\" (UniqueName: \"kubernetes.io/projected/6840207c-050c-432a-b36b-2becf3861d67-kube-api-access-gp95m\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:08.185859 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.185858    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6840207c-050c-432a-b36b-2becf3861d67-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:08.185990 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.185868    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6840207c-050c-432a-b36b-2becf3861d67-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:51:08.192146 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.192129    2560 scope.go:117] "RemoveContainer" containerID="610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783"
Jul 07 15:51:08.192361 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:51:08.192344    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783\": container with ID starting with 610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783 not found: ID does not exist" containerID="610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783"
Jul 07 15:51:08.192446 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.192370    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783"} err="failed to get container status \"610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783\": rpc error: code = NotFound desc = could not find container \"610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783\": container with ID starting with 610a5aedcc3d03344ddc265321efa27a23dde69e7b8987ec682e0a1af92ae783 not found: ID does not exist"
Jul 07 15:51:08.209191 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.209163    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"]
Jul 07 15:51:08.210911 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:08.210891    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/auth-test-model-458f34-kserve-7bfbfbc467-nzrm2"]
Jul 07 15:51:09.137868 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:09.137835    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6840207c-050c-432a-b36b-2becf3861d67" path="/var/lib/kubelet/pods/6840207c-050c-432a-b36b-2becf3861d67/volumes"
Jul 07 15:51:09.138277 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:09.138218    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78fede4a-7415-4414-ac2c-f0e8584067c4" path="/var/lib/kubelet/pods/78fede4a-7415-4414-ac2c-f0e8584067c4/volumes"
Jul 07 15:51:27.725671 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.725630    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"]
Jul 07 15:51:27.726124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.726000    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fede4a-7415-4414-ac2c-f0e8584067c4" containerName="main"
Jul 07 15:51:27.726124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.726013    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fede4a-7415-4414-ac2c-f0e8584067c4" containerName="main"
Jul 07 15:51:27.726124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.726034    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6840207c-050c-432a-b36b-2becf3861d67" containerName="main"
Jul 07 15:51:27.726124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.726039    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6840207c-050c-432a-b36b-2becf3861d67" containerName="main"
Jul 07 15:51:27.726124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.726116    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78fede4a-7415-4414-ac2c-f0e8584067c4" containerName="main"
Jul 07 15:51:27.726124 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.726127    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6840207c-050c-432a-b36b-2becf3861d67" containerName="main"
Jul 07 15:51:27.730427 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.730411    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.734349 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.734325    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"default-dockercfg-n5tf6\""
Jul 07 15:51:27.735151 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.735136    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"openshift-service-ca.crt\""
Jul 07 15:51:27.736749 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.736685    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"kube-root-ca.crt\""
Jul 07 15:51:27.737221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.737199    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"sub-test-model-458f34-kserve-self-signed-certs\""
Jul 07 15:51:27.755222 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.755196    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"]
Jul 07 15:51:27.869170 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.869134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-dshm\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.869348 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.869177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tmp-dir\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.869348 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.869210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tls-certs\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.869463 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.869345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-model-cache\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.869463 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.869374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhdg\" (UniqueName: \"kubernetes.io/projected/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-kube-api-access-bdhdg\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.869463 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.869427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-home\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970026 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.969988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-model-cache\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970026 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhdg\" (UniqueName: \"kubernetes.io/projected/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-kube-api-access-bdhdg\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970272 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-home\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970272 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-dshm\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970382 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tmp-dir\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970480 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970450    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-home\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970537 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970498    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-model-cache\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970635 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tls-certs\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.970781 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.970658    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tmp-dir\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.972441 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.972422    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-dshm\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.972709 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.972690    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tls-certs\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:27.988927 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:27.988850    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhdg\" (UniqueName: \"kubernetes.io/projected/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-kube-api-access-bdhdg\") pod \"sub-test-model-458f34-kserve-6d75f7dc55-nd6pm\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") " pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:28.039050 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:28.039008    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:28.196703 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:28.196668    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"]
Jul 07 15:51:28.201516 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:51:28.201484    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f594540_bd5a_4b10_94cc_d5b0cc937fd3.slice/crio-ceb63f1ebdd59258ee3a04b5da5c37f63b2274ab5aaa510531cc2c8b02f22c6c WatchSource:0}: Error finding container ceb63f1ebdd59258ee3a04b5da5c37f63b2274ab5aaa510531cc2c8b02f22c6c: Status 404 returned error can't find the container with id ceb63f1ebdd59258ee3a04b5da5c37f63b2274ab5aaa510531cc2c8b02f22c6c
Jul 07 15:51:28.276345 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:28.276309    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm" event={"ID":"8f594540-bd5a-4b10-94cc-d5b0cc937fd3","Type":"ContainerStarted","Data":"fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe"}
Jul 07 15:51:28.276520 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:28.276361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm" event={"ID":"8f594540-bd5a-4b10-94cc-d5b0cc937fd3","Type":"ContainerStarted","Data":"ceb63f1ebdd59258ee3a04b5da5c37f63b2274ab5aaa510531cc2c8b02f22c6c"}
Jul 07 15:51:28.375322 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:28.375262    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm" podStartSLOduration=1.375240507 podStartE2EDuration="1.375240507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:28.365911897 +0000 UTC m=+2305.774020799" watchObservedRunningTime="2026-07-07 15:51:28.375240507 +0000 UTC m=+2305.783349409"
Jul 07 15:51:38.039621 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:38.039579    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:38.039621 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:38.039628    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:38.052304 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:38.052272    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:51:38.335040 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:51:38.334959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:52:06.074131 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:06.074092    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"]
Jul 07 15:52:09.137481 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.137454    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"]
Jul 07 15:52:09.140952 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.140930    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.143775 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.143758    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"kube-root-ca.crt\""
Jul 07 15:52:09.143906 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.143793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"openshift-service-ca.crt\""
Jul 07 15:52:09.144725 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.144711    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"default-dockercfg-7g94j\""
Jul 07 15:52:09.144820 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.144783    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1f2ee2\"/\"rate-test-model-1f2ee2-kserve-self-signed-certs\""
Jul 07 15:52:09.153009 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.152989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"]
Jul 07 15:52:09.236227 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.236184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-home\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.236425 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.236244    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae4df1f-5e6f-4967-9834-aba24048f564-tls-certs\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.236425 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.236357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-model-cache\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.236526 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.236438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-tmp-dir\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.236526 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.236481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-dshm\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.236600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.236539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmhj\" (UniqueName: \"kubernetes.io/projected/8ae4df1f-5e6f-4967-9834-aba24048f564-kube-api-access-bdmhj\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.337647 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.337615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae4df1f-5e6f-4967-9834-aba24048f564-tls-certs\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.337845 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.337671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-model-cache\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.337845 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.337705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-tmp-dir\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.337845 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.337739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-dshm\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.338004 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.337836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmhj\" (UniqueName: \"kubernetes.io/projected/8ae4df1f-5e6f-4967-9834-aba24048f564-kube-api-access-bdmhj\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.338004 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.337920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-home\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.338122 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.338098    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-model-cache\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.338181 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.338166    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-tmp-dir\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.338278 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.338256    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-home\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.340017 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.339988    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-dshm\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.340261 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.340244    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae4df1f-5e6f-4967-9834-aba24048f564-tls-certs\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.348556 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.348537    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmhj\" (UniqueName: \"kubernetes.io/projected/8ae4df1f-5e6f-4967-9834-aba24048f564-kube-api-access-bdmhj\") pod \"rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") " pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.450664 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.450573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:09.594530 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:09.594500    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"]
Jul 07 15:52:09.597113 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:52:09.597086    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ae4df1f_5e6f_4967_9834_aba24048f564.slice/crio-9cc75e31d30871f6fc0eb9b8a3cbc04bd7c311136fa25fede102428ef45b9a5e WatchSource:0}: Error finding container 9cc75e31d30871f6fc0eb9b8a3cbc04bd7c311136fa25fede102428ef45b9a5e: Status 404 returned error can't find the container with id 9cc75e31d30871f6fc0eb9b8a3cbc04bd7c311136fa25fede102428ef45b9a5e
Jul 07 15:52:10.476917 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:10.476879    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x" event={"ID":"8ae4df1f-5e6f-4967-9834-aba24048f564","Type":"ContainerStarted","Data":"fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d"}
Jul 07 15:52:10.477442 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:10.476932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x" event={"ID":"8ae4df1f-5e6f-4967-9834-aba24048f564","Type":"ContainerStarted","Data":"9cc75e31d30871f6fc0eb9b8a3cbc04bd7c311136fa25fede102428ef45b9a5e"}
Jul 07 15:52:10.496944 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:10.496264    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x" podStartSLOduration=1.496243813 podStartE2EDuration="1.496243813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:10.495745789 +0000 UTC m=+2347.903854695" watchObservedRunningTime="2026-07-07 15:52:10.496243813 +0000 UTC m=+2347.904352718"
Jul 07 15:52:19.450923 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:19.450884    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:19.451318 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:19.450942    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:19.463635 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:19.463606    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:19.530620 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:19.530586    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:52:21.094059 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.094011    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm" podUID="8f594540-bd5a-4b10-94cc-d5b0cc937fd3" containerName="main" containerID="cri-o://fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe" gracePeriod=45
Jul 07 15:52:21.346928 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.346861    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:52:21.350033 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350013    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-dshm\") pod \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") "
Jul 07 15:52:21.350174 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350065    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tls-certs\") pod \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") "
Jul 07 15:52:21.350174 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350096    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-model-cache\") pod \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") "
Jul 07 15:52:21.350174 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350130    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdhdg\" (UniqueName: \"kubernetes.io/projected/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-kube-api-access-bdhdg\") pod \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") "
Jul 07 15:52:21.350174 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350158    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tmp-dir\") pod \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") "
Jul 07 15:52:21.350421 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350182    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-home\") pod \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\" (UID: \"8f594540-bd5a-4b10-94cc-d5b0cc937fd3\") "
Jul 07 15:52:21.350421 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350352    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f594540-bd5a-4b10-94cc-d5b0cc937fd3" (UID: "8f594540-bd5a-4b10-94cc-d5b0cc937fd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:52:21.350553 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350517    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:52:21.350610 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350564    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-home" (OuterVolumeSpecName: "home") pod "8f594540-bd5a-4b10-94cc-d5b0cc937fd3" (UID: "8f594540-bd5a-4b10-94cc-d5b0cc937fd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:52:21.350845 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.350809    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f594540-bd5a-4b10-94cc-d5b0cc937fd3" (UID: "8f594540-bd5a-4b10-94cc-d5b0cc937fd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:52:21.352409 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.352347    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-dshm" (OuterVolumeSpecName: "dshm") pod "8f594540-bd5a-4b10-94cc-d5b0cc937fd3" (UID: "8f594540-bd5a-4b10-94cc-d5b0cc937fd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:52:21.352565 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.352421    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f594540-bd5a-4b10-94cc-d5b0cc937fd3" (UID: "8f594540-bd5a-4b10-94cc-d5b0cc937fd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:52:21.352565 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.352434    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-kube-api-access-bdhdg" (OuterVolumeSpecName: "kube-api-access-bdhdg") pod "8f594540-bd5a-4b10-94cc-d5b0cc937fd3" (UID: "8f594540-bd5a-4b10-94cc-d5b0cc937fd3"). InnerVolumeSpecName "kube-api-access-bdhdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:52:21.451216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.451182    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:52:21.451216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.451210    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdhdg\" (UniqueName: \"kubernetes.io/projected/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-kube-api-access-bdhdg\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:52:21.451216 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.451221    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:52:21.451491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.451229    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:52:21.451491 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.451237    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f594540-bd5a-4b10-94cc-d5b0cc937fd3-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:52:21.529069 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.529031    2560 generic.go:356] "Generic (PLEG): container finished" podID="8f594540-bd5a-4b10-94cc-d5b0cc937fd3" containerID="fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe" exitCode=0
Jul 07 15:52:21.529221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.529080    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"
Jul 07 15:52:21.529221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.529121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm" event={"ID":"8f594540-bd5a-4b10-94cc-d5b0cc937fd3","Type":"ContainerDied","Data":"fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe"}
Jul 07 15:52:21.529221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.529167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm" event={"ID":"8f594540-bd5a-4b10-94cc-d5b0cc937fd3","Type":"ContainerDied","Data":"ceb63f1ebdd59258ee3a04b5da5c37f63b2274ab5aaa510531cc2c8b02f22c6c"}
Jul 07 15:52:21.529221 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.529185    2560 scope.go:117] "RemoveContainer" containerID="fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe"
Jul 07 15:52:21.538543 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.538520    2560 scope.go:117] "RemoveContainer" containerID="fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe"
Jul 07 15:52:21.538787 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:52:21.538768    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe\": container with ID starting with fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe not found: ID does not exist" containerID="fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe"
Jul 07 15:52:21.538830 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.538796    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe"} err="failed to get container status \"fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe\": rpc error: code = NotFound desc = could not find container \"fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe\": container with ID starting with fc4d76e0f299833cac325123aa89923ba612d2756b840ee0a56c5934defc14fe not found: ID does not exist"
Jul 07 15:52:21.551374 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.551343    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"]
Jul 07 15:52:21.555353 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:21.555328    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/sub-test-model-458f34-kserve-6d75f7dc55-nd6pm"]
Jul 07 15:52:23.137019 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:23.136985    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f594540-bd5a-4b10-94cc-d5b0cc937fd3" path="/var/lib/kubelet/pods/8f594540-bd5a-4b10-94cc-d5b0cc937fd3/volumes"
Jul 07 15:52:50.340335 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.340302    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"]
Jul 07 15:52:50.340813 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.340771    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f594540-bd5a-4b10-94cc-d5b0cc937fd3" containerName="main"
Jul 07 15:52:50.340813 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.340786    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f594540-bd5a-4b10-94cc-d5b0cc937fd3" containerName="main"
Jul 07 15:52:50.340888 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.340867    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f594540-bd5a-4b10-94cc-d5b0cc937fd3" containerName="main"
Jul 07 15:52:50.352665 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.352638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.355635 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.355611    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"kube-root-ca.crt\""
Jul 07 15:52:50.356251 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.356232    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"default-dockercfg-n5tf6\""
Jul 07 15:52:50.356541 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.356236    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"openshift-service-ca.crt\""
Jul 07 15:52:50.356974 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.356955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-458f34\"/\"rate-test-model-458f34-kserve-self-signed-certs\""
Jul 07 15:52:50.357990 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.357971    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"]
Jul 07 15:52:50.513682 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.513646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25lps\" (UniqueName: \"kubernetes.io/projected/f69661c1-5758-42bb-8b30-38da70054db5-kube-api-access-25lps\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.513865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.513749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-model-cache\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.513865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.513790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69661c1-5758-42bb-8b30-38da70054db5-tls-certs\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.513865 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.513855    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-dshm\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.513990 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.513933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-tmp-dir\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.513990 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.513966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-home\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.614869 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.614771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-dshm\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.614869 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.614830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-tmp-dir\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.614869 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.614867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-home\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.615163 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.614888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25lps\" (UniqueName: \"kubernetes.io/projected/f69661c1-5758-42bb-8b30-38da70054db5-kube-api-access-25lps\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.615163 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.614948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-model-cache\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.615163 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.614978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69661c1-5758-42bb-8b30-38da70054db5-tls-certs\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.615332 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.615313    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-home\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.615439 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.615417    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-model-cache\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.615506 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.615441    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-tmp-dir\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.617102 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.617078    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-dshm\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.617456 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.617438    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69661c1-5758-42bb-8b30-38da70054db5-tls-certs\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.625286 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.625260    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25lps\" (UniqueName: \"kubernetes.io/projected/f69661c1-5758-42bb-8b30-38da70054db5-kube-api-access-25lps\") pod \"rate-test-model-458f34-kserve-7879ddc859-6rzn7\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") " pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.664254 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.664221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:52:50.802479 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:50.802452    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"]
Jul 07 15:52:50.804587 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:52:50.804558    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf69661c1_5758_42bb_8b30_38da70054db5.slice/crio-b9b1f180bd69726786c152c472641f4e6d37bf68d69fc911d6330970697a26a5 WatchSource:0}: Error finding container b9b1f180bd69726786c152c472641f4e6d37bf68d69fc911d6330970697a26a5: Status 404 returned error can't find the container with id b9b1f180bd69726786c152c472641f4e6d37bf68d69fc911d6330970697a26a5
Jul 07 15:52:51.670792 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:51.670758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7" event={"ID":"f69661c1-5758-42bb-8b30-38da70054db5","Type":"ContainerStarted","Data":"41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf"}
Jul 07 15:52:51.670792 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:51.670797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7" event={"ID":"f69661c1-5758-42bb-8b30-38da70054db5","Type":"ContainerStarted","Data":"b9b1f180bd69726786c152c472641f4e6d37bf68d69fc911d6330970697a26a5"}
Jul 07 15:52:51.690613 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:52:51.690565    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7" podStartSLOduration=1.690549384 podStartE2EDuration="1.690549384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:51.68904352 +0000 UTC m=+2389.097152434" watchObservedRunningTime="2026-07-07 15:52:51.690549384 +0000 UTC m=+2389.098658285"
Jul 07 15:53:00.665142 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:00.665101    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:53:00.665579 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:00.665156    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:53:00.677431 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:00.677376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:53:00.727695 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:00.727666    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:53:32.287119 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:32.287041    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"]
Jul 07 15:53:33.259782 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:33.259749    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"]
Jul 07 15:53:35.144407 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.144366    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"]
Jul 07 15:53:35.144881 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.144581    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66" podUID="b1852bed-917c-469d-8c16-813af31f5079" containerName="maas-api" containerID="cri-o://d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705" gracePeriod=30
Jul 07 15:53:35.590233 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.590205    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:53:35.738612 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.738525    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1852bed-917c-469d-8c16-813af31f5079-openshift-service-ca\") pod \"b1852bed-917c-469d-8c16-813af31f5079\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") "
Jul 07 15:53:35.738612 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.738573    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls\") pod \"b1852bed-917c-469d-8c16-813af31f5079\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") "
Jul 07 15:53:35.738924 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.738636    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnzzt\" (UniqueName: \"kubernetes.io/projected/b1852bed-917c-469d-8c16-813af31f5079-kube-api-access-xnzzt\") pod \"b1852bed-917c-469d-8c16-813af31f5079\" (UID: \"b1852bed-917c-469d-8c16-813af31f5079\") "
Jul 07 15:53:35.738924 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.738881    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1852bed-917c-469d-8c16-813af31f5079-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1852bed-917c-469d-8c16-813af31f5079" (UID: "b1852bed-917c-469d-8c16-813af31f5079"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:53:35.740736 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.740706    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1852bed-917c-469d-8c16-813af31f5079-kube-api-access-xnzzt" (OuterVolumeSpecName: "kube-api-access-xnzzt") pod "b1852bed-917c-469d-8c16-813af31f5079" (UID: "b1852bed-917c-469d-8c16-813af31f5079"). InnerVolumeSpecName "kube-api-access-xnzzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:53:35.740841 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.740735    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1852bed-917c-469d-8c16-813af31f5079" (UID: "b1852bed-917c-469d-8c16-813af31f5079"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:53:35.839806 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.839767    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1852bed-917c-469d-8c16-813af31f5079-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:35.839806 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.839799    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnzzt\" (UniqueName: \"kubernetes.io/projected/b1852bed-917c-469d-8c16-813af31f5079-kube-api-access-xnzzt\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:35.839806 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.839810    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1852bed-917c-469d-8c16-813af31f5079-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:35.876765 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.876730    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1852bed-917c-469d-8c16-813af31f5079" containerID="d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705" exitCode=0
Jul 07 15:53:35.876938 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.876778    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"
Jul 07 15:53:35.876938 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.876787    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66" event={"ID":"b1852bed-917c-469d-8c16-813af31f5079","Type":"ContainerDied","Data":"d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705"}
Jul 07 15:53:35.876938 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.876838    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66" event={"ID":"b1852bed-917c-469d-8c16-813af31f5079","Type":"ContainerDied","Data":"98a921242313ad85fd467dc1c582a555301ae3f2d3817d1fb64ea16bd1c11878"}
Jul 07 15:53:35.876938 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.876867    2560 scope.go:117] "RemoveContainer" containerID="d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705"
Jul 07 15:53:35.886159 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.886139    2560 scope.go:117] "RemoveContainer" containerID="d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705"
Jul 07 15:53:35.886427 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:53:35.886409    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705\": container with ID starting with d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705 not found: ID does not exist" containerID="d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705"
Jul 07 15:53:35.886483 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.886437    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705"} err="failed to get container status \"d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705\": rpc error: code = NotFound desc = could not find container \"d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705\": container with ID starting with d006a931c1f5dc56e4a553ebb02672156a225151a99dc10431488d973a991705 not found: ID does not exist"
Jul 07 15:53:35.898207 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.898184    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"]
Jul 07 15:53:35.904076 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:35.904056    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-1f2ee2-dc66b49fc-tmj66"]
Jul 07 15:53:37.137225 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:37.137181    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1852bed-917c-469d-8c16-813af31f5079" path="/var/lib/kubelet/pods/b1852bed-917c-469d-8c16-813af31f5079/volumes"
Jul 07 15:53:47.307309 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.307261    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x" podUID="8ae4df1f-5e6f-4967-9834-aba24048f564" containerName="main" containerID="cri-o://fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d" gracePeriod=45
Jul 07 15:53:47.570222 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.570153    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:53:47.650105 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650058    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-model-cache\") pod \"8ae4df1f-5e6f-4967-9834-aba24048f564\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") "
Jul 07 15:53:47.650284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650153    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-tmp-dir\") pod \"8ae4df1f-5e6f-4967-9834-aba24048f564\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") "
Jul 07 15:53:47.650284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-home\") pod \"8ae4df1f-5e6f-4967-9834-aba24048f564\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") "
Jul 07 15:53:47.650284 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650235    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-dshm\") pod \"8ae4df1f-5e6f-4967-9834-aba24048f564\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") "
Jul 07 15:53:47.650487 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650294    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdmhj\" (UniqueName: \"kubernetes.io/projected/8ae4df1f-5e6f-4967-9834-aba24048f564-kube-api-access-bdmhj\") pod \"8ae4df1f-5e6f-4967-9834-aba24048f564\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") "
Jul 07 15:53:47.650487 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650343    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae4df1f-5e6f-4967-9834-aba24048f564-tls-certs\") pod \"8ae4df1f-5e6f-4967-9834-aba24048f564\" (UID: \"8ae4df1f-5e6f-4967-9834-aba24048f564\") "
Jul 07 15:53:47.650487 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650375    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ae4df1f-5e6f-4967-9834-aba24048f564" (UID: "8ae4df1f-5e6f-4967-9834-aba24048f564"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:47.650487 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650413    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ae4df1f-5e6f-4967-9834-aba24048f564" (UID: "8ae4df1f-5e6f-4967-9834-aba24048f564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:47.650487 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650447    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-home" (OuterVolumeSpecName: "home") pod "8ae4df1f-5e6f-4967-9834-aba24048f564" (UID: "8ae4df1f-5e6f-4967-9834-aba24048f564"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:47.650720 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650648    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:47.650720 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650667    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:47.650720 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.650683    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:47.652472 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.652447    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-dshm" (OuterVolumeSpecName: "dshm") pod "8ae4df1f-5e6f-4967-9834-aba24048f564" (UID: "8ae4df1f-5e6f-4967-9834-aba24048f564"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:47.652561 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.652486    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae4df1f-5e6f-4967-9834-aba24048f564-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ae4df1f-5e6f-4967-9834-aba24048f564" (UID: "8ae4df1f-5e6f-4967-9834-aba24048f564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:53:47.652600 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.652554    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae4df1f-5e6f-4967-9834-aba24048f564-kube-api-access-bdmhj" (OuterVolumeSpecName: "kube-api-access-bdmhj") pod "8ae4df1f-5e6f-4967-9834-aba24048f564" (UID: "8ae4df1f-5e6f-4967-9834-aba24048f564"). InnerVolumeSpecName "kube-api-access-bdmhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:53:47.751816 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.751773    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae4df1f-5e6f-4967-9834-aba24048f564-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:47.751816 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.751808    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmhj\" (UniqueName: \"kubernetes.io/projected/8ae4df1f-5e6f-4967-9834-aba24048f564-kube-api-access-bdmhj\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:47.751816 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.751819    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae4df1f-5e6f-4967-9834-aba24048f564-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:47.930545 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.930512    2560 generic.go:356] "Generic (PLEG): container finished" podID="8ae4df1f-5e6f-4967-9834-aba24048f564" containerID="fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d" exitCode=0
Jul 07 15:53:47.930729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.930557    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"
Jul 07 15:53:47.930729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.930578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x" event={"ID":"8ae4df1f-5e6f-4967-9834-aba24048f564","Type":"ContainerDied","Data":"fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d"}
Jul 07 15:53:47.930729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.930611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x" event={"ID":"8ae4df1f-5e6f-4967-9834-aba24048f564","Type":"ContainerDied","Data":"9cc75e31d30871f6fc0eb9b8a3cbc04bd7c311136fa25fede102428ef45b9a5e"}
Jul 07 15:53:47.930729 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.930627    2560 scope.go:117] "RemoveContainer" containerID="fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d"
Jul 07 15:53:47.938934 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.938917    2560 scope.go:117] "RemoveContainer" containerID="fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d"
Jul 07 15:53:47.939199 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:53:47.939182    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d\": container with ID starting with fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d not found: ID does not exist" containerID="fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d"
Jul 07 15:53:47.939250 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.939208    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d"} err="failed to get container status \"fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d\": rpc error: code = NotFound desc = could not find container \"fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d\": container with ID starting with fbf13bfcd5a9bdc46b7f95183605796907b305caeb6681d3ef9a0088c853800d not found: ID does not exist"
Jul 07 15:53:47.951422 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.951400    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"]
Jul 07 15:53:47.955174 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:47.955152    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1f2ee2/rate-test-model-1f2ee2-kserve-86c5496ff-hmq6x"]
Jul 07 15:53:48.279369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.279279    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7" podUID="f69661c1-5758-42bb-8b30-38da70054db5" containerName="main" containerID="cri-o://41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf" gracePeriod=45
Jul 07 15:53:48.532228 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.532161    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:53:48.558953 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.558924    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25lps\" (UniqueName: \"kubernetes.io/projected/f69661c1-5758-42bb-8b30-38da70054db5-kube-api-access-25lps\") pod \"f69661c1-5758-42bb-8b30-38da70054db5\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") "
Jul 07 15:53:48.559125 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.558982    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69661c1-5758-42bb-8b30-38da70054db5-tls-certs\") pod \"f69661c1-5758-42bb-8b30-38da70054db5\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") "
Jul 07 15:53:48.559125 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.559015    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-dshm\") pod \"f69661c1-5758-42bb-8b30-38da70054db5\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") "
Jul 07 15:53:48.559125 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.559036    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-model-cache\") pod \"f69661c1-5758-42bb-8b30-38da70054db5\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") "
Jul 07 15:53:48.559125 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.559053    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-home\") pod \"f69661c1-5758-42bb-8b30-38da70054db5\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") "
Jul 07 15:53:48.559125 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.559075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-tmp-dir\") pod \"f69661c1-5758-42bb-8b30-38da70054db5\" (UID: \"f69661c1-5758-42bb-8b30-38da70054db5\") "
Jul 07 15:53:48.559417 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.559342    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-home" (OuterVolumeSpecName: "home") pod "f69661c1-5758-42bb-8b30-38da70054db5" (UID: "f69661c1-5758-42bb-8b30-38da70054db5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:48.559492 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.559414    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f69661c1-5758-42bb-8b30-38da70054db5" (UID: "f69661c1-5758-42bb-8b30-38da70054db5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:48.559492 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.559453    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f69661c1-5758-42bb-8b30-38da70054db5" (UID: "f69661c1-5758-42bb-8b30-38da70054db5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:48.561126 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.561101    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69661c1-5758-42bb-8b30-38da70054db5-kube-api-access-25lps" (OuterVolumeSpecName: "kube-api-access-25lps") pod "f69661c1-5758-42bb-8b30-38da70054db5" (UID: "f69661c1-5758-42bb-8b30-38da70054db5"). InnerVolumeSpecName "kube-api-access-25lps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:53:48.561231 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.561109    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-dshm" (OuterVolumeSpecName: "dshm") pod "f69661c1-5758-42bb-8b30-38da70054db5" (UID: "f69661c1-5758-42bb-8b30-38da70054db5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:53:48.561231 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.561181    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69661c1-5758-42bb-8b30-38da70054db5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f69661c1-5758-42bb-8b30-38da70054db5" (UID: "f69661c1-5758-42bb-8b30-38da70054db5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:53:48.659856 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.659815    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-dshm\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:48.659856 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.659849    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-model-cache\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:48.659856 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.659862    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-home\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:48.660109 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.659873    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69661c1-5758-42bb-8b30-38da70054db5-tmp-dir\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:48.660109 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.659884    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25lps\" (UniqueName: \"kubernetes.io/projected/f69661c1-5758-42bb-8b30-38da70054db5-kube-api-access-25lps\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:48.660109 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.659895    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69661c1-5758-42bb-8b30-38da70054db5-tls-certs\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:53:48.937712 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.937676    2560 generic.go:356] "Generic (PLEG): container finished" podID="f69661c1-5758-42bb-8b30-38da70054db5" containerID="41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf" exitCode=0
Jul 07 15:53:48.937887 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.937720    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"
Jul 07 15:53:48.937887 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.937749    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7" event={"ID":"f69661c1-5758-42bb-8b30-38da70054db5","Type":"ContainerDied","Data":"41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf"}
Jul 07 15:53:48.937887 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.937779    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7" event={"ID":"f69661c1-5758-42bb-8b30-38da70054db5","Type":"ContainerDied","Data":"b9b1f180bd69726786c152c472641f4e6d37bf68d69fc911d6330970697a26a5"}
Jul 07 15:53:48.937887 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.937795    2560 scope.go:117] "RemoveContainer" containerID="41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf"
Jul 07 15:53:48.946550 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.946527    2560 scope.go:117] "RemoveContainer" containerID="41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf"
Jul 07 15:53:48.946814 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:53:48.946795    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf\": container with ID starting with 41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf not found: ID does not exist" containerID="41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf"
Jul 07 15:53:48.946875 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.946823    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf"} err="failed to get container status \"41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf\": rpc error: code = NotFound desc = could not find container \"41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf\": container with ID starting with 41fd597db49c4aa20d9b50c5cad63e9dd1f494d8ee170e51d6a86c7967aeb7bf not found: ID does not exist"
Jul 07 15:53:48.959545 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.959521    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"]
Jul 07 15:53:48.965511 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:48.965484    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-458f34/rate-test-model-458f34-kserve-7879ddc859-6rzn7"]
Jul 07 15:53:49.137629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:49.137599    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae4df1f-5e6f-4967-9834-aba24048f564" path="/var/lib/kubelet/pods/8ae4df1f-5e6f-4967-9834-aba24048f564/volumes"
Jul 07 15:53:49.137988 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:53:49.137975    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f69661c1-5758-42bb-8b30-38da70054db5" path="/var/lib/kubelet/pods/f69661c1-5758-42bb-8b30-38da70054db5/volumes"
Jul 07 15:54:18.756594 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:18.756557    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"]
Jul 07 15:54:18.757061 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:18.756840    2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7" podUID="55368b3e-2ec0-40ce-836d-5a4f37e06bf7" containerName="maas-api" containerID="cri-o://44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a" gracePeriod=30
Jul 07 15:54:19.001629 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.001603    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:54:19.075844 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.075755    2560 generic.go:356] "Generic (PLEG): container finished" podID="55368b3e-2ec0-40ce-836d-5a4f37e06bf7" containerID="44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a" exitCode=0
Jul 07 15:54:19.075844 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.075793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7" event={"ID":"55368b3e-2ec0-40ce-836d-5a4f37e06bf7","Type":"ContainerDied","Data":"44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a"}
Jul 07 15:54:19.075844 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.075798    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"
Jul 07 15:54:19.075844 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.075830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7" event={"ID":"55368b3e-2ec0-40ce-836d-5a4f37e06bf7","Type":"ContainerDied","Data":"f1867b4f8852a57695c35e58659be2ae563297714a9469679000f864fcd13c3d"}
Jul 07 15:54:19.076114 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.075851    2560 scope.go:117] "RemoveContainer" containerID="44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a"
Jul 07 15:54:19.083951 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.083930    2560 scope.go:117] "RemoveContainer" containerID="44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a"
Jul 07 15:54:19.084215 ip-10-0-131-161 kubenswrapper[2560]: E0707 15:54:19.084192    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a\": container with ID starting with 44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a not found: ID does not exist" containerID="44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a"
Jul 07 15:54:19.084281 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.084229    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a"} err="failed to get container status \"44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a\": rpc error: code = NotFound desc = could not find container \"44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a\": container with ID starting with 44af9c96713c8832cbe524afb926d88b086b53e6f0666a4c626f344cf473e98a not found: ID does not exist"
Jul 07 15:54:19.132171 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.132141    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls\") pod \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") "
Jul 07 15:54:19.132300 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.132209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-openshift-service-ca\") pod \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") "
Jul 07 15:54:19.132300 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.132275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp6z2\" (UniqueName: \"kubernetes.io/projected/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-kube-api-access-xp6z2\") pod \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\" (UID: \"55368b3e-2ec0-40ce-836d-5a4f37e06bf7\") "
Jul 07 15:54:19.132594 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.132571    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55368b3e-2ec0-40ce-836d-5a4f37e06bf7" (UID: "55368b3e-2ec0-40ce-836d-5a4f37e06bf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:54:19.134338 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.134305    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55368b3e-2ec0-40ce-836d-5a4f37e06bf7" (UID: "55368b3e-2ec0-40ce-836d-5a4f37e06bf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:54:19.134529 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.134511    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-kube-api-access-xp6z2" (OuterVolumeSpecName: "kube-api-access-xp6z2") pod "55368b3e-2ec0-40ce-836d-5a4f37e06bf7" (UID: "55368b3e-2ec0-40ce-836d-5a4f37e06bf7"). InnerVolumeSpecName "kube-api-access-xp6z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:54:19.233352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.233318    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp6z2\" (UniqueName: \"kubernetes.io/projected/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-kube-api-access-xp6z2\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:54:19.233352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.233347    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-maas-api-tls\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:54:19.233352 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.233364    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55368b3e-2ec0-40ce-836d-5a4f37e06bf7-openshift-service-ca\") on node \"ip-10-0-131-161.ec2.internal\" DevicePath \"\""
Jul 07 15:54:19.402241 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.402205    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"]
Jul 07 15:54:19.414371 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:19.414335    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-458f34-8c54f49-q6hg7"]
Jul 07 15:54:21.142568 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:21.142533    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55368b3e-2ec0-40ce-836d-5a4f37e06bf7" path="/var/lib/kubelet/pods/55368b3e-2ec0-40ce-836d-5a4f37e06bf7/volumes"
Jul 07 15:54:31.339565 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:31.339537    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5c8fbf54f8-6x7c5_6db1a320-c588-48e4-a74e-f3e2346ee645/maas-api/0.log"
Jul 07 15:54:31.688969 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:31.688938    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c85dbf754-xr2dp_04b812df-e722-4036-9a45-1f29267c50fa/manager/0.log"
Jul 07 15:54:32.173110 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:32.173082    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcbf69668-sf2w5_9ee9870b-660d-4abc-b940-0d35521a72e1/copy-manifests/0.log"
Jul 07 15:54:32.180688 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:32.180665    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcbf69668-sf2w5_9ee9870b-660d-4abc-b940-0d35521a72e1/manager/0.log"
Jul 07 15:54:32.424085 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:32.424003    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-h26ct_78a797b5-513d-471b-acaf-7950a41d5b71/postgres/0.log"
Jul 07 15:54:35.207550 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:35.207518    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6c759bc-8f6cn_42f80fde-5172-4e59-a730-544ec86df5bc/kube-auth-proxy/0.log"
Jul 07 15:54:35.438843 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:35.438809    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9r8rt_11c753c8-ff16-45fb-aa74-af8982415637/payload-pre-processing/0.log"
Jul 07 15:54:35.553699 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:35.553623    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-5g7hl_25bbea9b-4f7f-4e2d-ba7b-0064190894be/payload-processing/0.log"
Jul 07 15:54:36.007532 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:36.007500    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-87fjk_f0381424-fe57-40ed-8cdf-852666730b54/main/0.log"
Jul 07 15:54:36.367594 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:36.367518    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdk7d_7fe436ea-1efe-4841-8e72-b67cb132e2a4/main/0.log"
Jul 07 15:54:36.589555 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:36.589524    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-l2hqz_a00c2e63-b090-4886-90c8-81d96e52ef44/main/0.log"
Jul 07 15:54:43.683519 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:43.683488    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6njzf_247e63ad-de91-47f1-bae3-c697a7e5f702/global-pull-secret-syncer/0.log"
Jul 07 15:54:43.930919 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:43.930884    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kslrc_e95cd953-6426-481f-af3c-45c51721226f/konnectivity-agent/0.log"
Jul 07 15:54:43.970616 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:43.970540    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-161.ec2.internal_d6db363e1603ceda82be0f2b17302f46/haproxy/0.log"
Jul 07 15:54:51.937306 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:51.937274    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-hdf2t_dbd0fb21-7b50-4317-bc2c-e9a903f633f1/kube-state-metrics/0.log"
Jul 07 15:54:51.967126 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:51.967091    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-hdf2t_dbd0fb21-7b50-4317-bc2c-e9a903f633f1/kube-rbac-proxy-main/0.log"
Jul 07 15:54:52.003077 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.003043    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-hdf2t_dbd0fb21-7b50-4317-bc2c-e9a903f633f1/kube-rbac-proxy-self/0.log"
Jul 07 15:54:52.182939 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.182907    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nqx8f_20b906c3-d257-41e0-bdeb-a28002532276/node-exporter/0.log"
Jul 07 15:54:52.213002 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.212929    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nqx8f_20b906c3-d257-41e0-bdeb-a28002532276/kube-rbac-proxy/0.log"
Jul 07 15:54:52.236313 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.236284    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nqx8f_20b906c3-d257-41e0-bdeb-a28002532276/init-textfile/0.log"
Jul 07 15:54:52.452807 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.452776    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4955ebd-6d2c-4686-b712-055f09c0cbcf/prometheus/0.log"
Jul 07 15:54:52.476998 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.476931    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4955ebd-6d2c-4686-b712-055f09c0cbcf/config-reloader/0.log"
Jul 07 15:54:52.502900 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.502869    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4955ebd-6d2c-4686-b712-055f09c0cbcf/thanos-sidecar/0.log"
Jul 07 15:54:52.527237 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.527206    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4955ebd-6d2c-4686-b712-055f09c0cbcf/kube-rbac-proxy-web/0.log"
Jul 07 15:54:52.559710 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.559677    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4955ebd-6d2c-4686-b712-055f09c0cbcf/kube-rbac-proxy/0.log"
Jul 07 15:54:52.586008 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.585979    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4955ebd-6d2c-4686-b712-055f09c0cbcf/kube-rbac-proxy-thanos/0.log"
Jul 07 15:54:52.612457 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.612429    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4955ebd-6d2c-4686-b712-055f09c0cbcf/init-config-reloader/0.log"
Jul 07 15:54:52.658856 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.658824    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-clzvl_c5e1b6d5-0e85-4978-a3ab-21ed4158c03e/prometheus-operator/0.log"
Jul 07 15:54:52.699151 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:52.699123    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-clzvl_c5e1b6d5-0e85-4978-a3ab-21ed4158c03e/kube-rbac-proxy/0.log"
Jul 07 15:54:53.621579 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621538    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"]
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621881    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae4df1f-5e6f-4967-9834-aba24048f564" containerName="main"
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621893    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae4df1f-5e6f-4967-9834-aba24048f564" containerName="main"
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621901    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f69661c1-5758-42bb-8b30-38da70054db5" containerName="main"
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621906    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f69661c1-5758-42bb-8b30-38da70054db5" containerName="main"
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621924    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55368b3e-2ec0-40ce-836d-5a4f37e06bf7" containerName="maas-api"
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621930    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="55368b3e-2ec0-40ce-836d-5a4f37e06bf7" containerName="maas-api"
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621937    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1852bed-917c-469d-8c16-813af31f5079" containerName="maas-api"
Jul 07 15:54:53.621962 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621943    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1852bed-917c-469d-8c16-813af31f5079" containerName="maas-api"
Jul 07 15:54:53.622211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621990    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="55368b3e-2ec0-40ce-836d-5a4f37e06bf7" containerName="maas-api"
Jul 07 15:54:53.622211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.621999    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1852bed-917c-469d-8c16-813af31f5079" containerName="maas-api"
Jul 07 15:54:53.622211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.622006    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae4df1f-5e6f-4967-9834-aba24048f564" containerName="main"
Jul 07 15:54:53.622211 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.622012    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f69661c1-5758-42bb-8b30-38da70054db5" containerName="main"
Jul 07 15:54:53.624708 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.624686    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.627317 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.627295    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wvtd8\"/\"default-dockercfg-4d48b\""
Jul 07 15:54:53.627480 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.627460    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvtd8\"/\"openshift-service-ca.crt\""
Jul 07 15:54:53.628256 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.628237    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvtd8\"/\"kube-root-ca.crt\""
Jul 07 15:54:53.637701 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.637679    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"]
Jul 07 15:54:53.732191 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.732147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-sys\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.732191 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.732187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-lib-modules\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.732493 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.732226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-podres\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.732493 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.732309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-proc\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.732493 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.732342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5sl\" (UniqueName: \"kubernetes.io/projected/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-kube-api-access-hc5sl\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833164 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-lib-modules\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-podres\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-proc\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5sl\" (UniqueName: \"kubernetes.io/projected/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-kube-api-access-hc5sl\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833312    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-lib-modules\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833324    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-proc\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-sys\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833354    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-podres\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.833369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.833364    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-sys\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.852950 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.852919    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5sl\" (UniqueName: \"kubernetes.io/projected/35c28273-8cf7-4ea0-b1ef-3c61d70b1b04-kube-api-access-hc5sl\") pod \"perf-node-gather-daemonset-pj599\" (UID: \"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04\") " pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:53.936217 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:53.936187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:54.088970 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:54.088938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"]
Jul 07 15:54:54.092250 ip-10-0-131-161 kubenswrapper[2560]: W0707 15:54:54.092220    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35c28273_8cf7_4ea0_b1ef_3c61d70b1b04.slice/crio-cea5664f409c02154f7348ec0dab7851f589fbfca3b9983cce75970acc985848 WatchSource:0}: Error finding container cea5664f409c02154f7348ec0dab7851f589fbfca3b9983cce75970acc985848: Status 404 returned error can't find the container with id cea5664f409c02154f7348ec0dab7851f589fbfca3b9983cce75970acc985848
Jul 07 15:54:54.234476 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:54.234443    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599" event={"ID":"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04","Type":"ContainerStarted","Data":"2443a24ca1ef180e3d33f5c9a73d4c8bf27ab147a5b0a7008b8e6b2d4e6f2e2b"}
Jul 07 15:54:54.234654 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:54.234582    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:54:54.234654 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:54.234623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599" event={"ID":"35c28273-8cf7-4ea0-b1ef-3c61d70b1b04","Type":"ContainerStarted","Data":"cea5664f409c02154f7348ec0dab7851f589fbfca3b9983cce75970acc985848"}
Jul 07 15:54:54.253706 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:54.253658    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599" podStartSLOduration=1.253642895 podStartE2EDuration="1.253642895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:54:54.253148014 +0000 UTC m=+2511.661256916" watchObservedRunningTime="2026-07-07 15:54:54.253642895 +0000 UTC m=+2511.661751796"
Jul 07 15:54:55.484006 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:55.483964    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-9xbsw_80bb84da-c2ad-47db-a15f-1df528b69cb0/download-server/0.log"
Jul 07 15:54:57.251162 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:57.251101    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jkls_04e52544-b5d7-428e-b574-a22cd624da55/dns/0.log"
Jul 07 15:54:57.279548 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:57.279516    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jkls_04e52544-b5d7-428e-b574-a22cd624da55/kube-rbac-proxy/0.log"
Jul 07 15:54:57.423873 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:57.423845    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fjq24_220a5ee3-b52f-414a-8d29-7453eda2bfb9/dns-node-resolver/0.log"
Jul 07 15:54:58.235879 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:58.235854    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmfmg_5d0644a9-b118-45fa-9e14-77ee2cd8c242/node-ca/0.log"
Jul 07 15:54:59.600607 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:59.600565    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6c759bc-8f6cn_42f80fde-5172-4e59-a730-544ec86df5bc/kube-auth-proxy/0.log"
Jul 07 15:54:59.716369 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:59.716340    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9r8rt_11c753c8-ff16-45fb-aa74-af8982415637/payload-pre-processing/0.log"
Jul 07 15:54:59.767994 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:54:59.767960    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-5g7hl_25bbea9b-4f7f-4e2d-ba7b-0064190894be/payload-processing/0.log"
Jul 07 15:55:00.248273 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:00.248233    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wvtd8/perf-node-gather-daemonset-pj599"
Jul 07 15:55:00.654203 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:00.654177    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mtfhv_9ea178d1-089a-4394-b4ce-fcd5ae5f8965/serve-healthcheck-canary/0.log"
Jul 07 15:55:01.505445 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:01.505375    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmsrf_0d8f6c7c-257b-4bf4-adef-ca757ed4d908/kube-rbac-proxy/0.log"
Jul 07 15:55:01.544051 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:01.544025    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmsrf_0d8f6c7c-257b-4bf4-adef-ca757ed4d908/exporter/0.log"
Jul 07 15:55:01.569574 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:01.569545    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmsrf_0d8f6c7c-257b-4bf4-adef-ca757ed4d908/extractor/0.log"
Jul 07 15:55:04.497016 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:04.496973    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6c759bc-8f6cn_42f80fde-5172-4e59-a730-544ec86df5bc/kube-auth-proxy/0.log"
Jul 07 15:55:04.620131 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:04.620096    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9r8rt_11c753c8-ff16-45fb-aa74-af8982415637/payload-pre-processing/0.log"
Jul 07 15:55:04.766479 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:04.766353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-5g7hl_25bbea9b-4f7f-4e2d-ba7b-0064190894be/payload-processing/0.log"
Jul 07 15:55:04.783106 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:04.783075    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5c8fbf54f8-6x7c5_6db1a320-c588-48e4-a74e-f3e2346ee645/maas-api/0.log"
Jul 07 15:55:05.001175 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:05.001137    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c85dbf754-xr2dp_04b812df-e722-4036-9a45-1f29267c50fa/manager/0.log"
Jul 07 15:55:05.317285 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:05.317257    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcbf69668-sf2w5_9ee9870b-660d-4abc-b940-0d35521a72e1/manager/0.log"
Jul 07 15:55:05.355775 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:05.355697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcbf69668-sf2w5_9ee9870b-660d-4abc-b940-0d35521a72e1/copy-manifests/0.log"
Jul 07 15:55:05.573593 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:05.573566    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-h26ct_78a797b5-513d-471b-acaf-7950a41d5b71/postgres/0.log"
Jul 07 15:55:06.230543 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:06.230517    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c6c759bc-8f6cn_42f80fde-5172-4e59-a730-544ec86df5bc/kube-auth-proxy/0.log"
Jul 07 15:55:06.308849 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:06.308818    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9r8rt_11c753c8-ff16-45fb-aa74-af8982415637/payload-pre-processing/0.log"
Jul 07 15:55:06.347851 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:06.347817    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-5g7hl_25bbea9b-4f7f-4e2d-ba7b-0064190894be/payload-processing/0.log"
Jul 07 15:55:06.990253 ip-10-0-131-161 kubenswrapper[2560]: I0707 15:55:06.990217    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5ffd868d8b-jnf2r_88a46d28-1af0-48b4-a2a6-e32f72e0c237/manager/0.log"