Jul 20 09:56:29.255803 ip-10-0-137-140 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 09:56:29.636881 ip-10-0-137-140 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:29.636881 ip-10-0-137-140 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 09:56:29.636881 ip-10-0-137-140 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:29.636881 ip-10-0-137-140 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 09:56:29.636881 ip-10-0-137-140 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:29.637921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.637747    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 09:56:29.644396 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644373    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:29.644396 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644397    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644401    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644405    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644408    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644411    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644414    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644417    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644420    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644422    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644425    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644428    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644430    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644433    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644439    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644442    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644445    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644447    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644450    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644452    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644456    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:29.644468 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644458    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644461    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644463    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644466    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644469    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644471    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644474    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644477    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644502    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644506    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644509    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644512    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644515    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644518    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644520    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644523    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644537    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644540    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644542    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644545    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:29.644946 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644547    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644550    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644552    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644555    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644557    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644559    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644562    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644565    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644567    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644570    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644572    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644575    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644577    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644580    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644583    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644585    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644588    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644590    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644593    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:29.645735 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644597    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644601    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644603    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644606    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644608    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644611    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644613    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644616    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644619    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644621    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644625    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644628    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644631    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644633    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644636    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644639    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644641    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644644    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644646    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:29.646241 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644648    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644652    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644655    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644657    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644660    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644663    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644665    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644667    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644671    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644673    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644676    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644678    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644681    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644683    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644686    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644688    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644690    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644693    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644695    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:29.646699 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.644698    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:29.647623 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647604    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:29.647623 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647622    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:29.647623 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647628    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647632    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647636    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647640    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647644    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647648    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647651    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647656    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647661    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647665    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647669    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647672    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647676    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647680    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647684    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647687    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647691    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647696    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647700    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647707    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:29.647795 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647713    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647717    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647722    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647726    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647730    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647734    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647738    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647742    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647746    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647750    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647754    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647759    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647764    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647768    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647773    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647777    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647782    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647786    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647790    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647794    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:29.648674 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647800    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647804    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647808    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647812    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647816    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647820    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647825    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647829    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647834    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647838    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647842    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647846    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647850    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647854    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647858    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647862    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647867    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647871    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647875    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647879    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:29.649547 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647883    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647887    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647891    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647895    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647899    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647903    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647908    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647913    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647918    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647922    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647926    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647933    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647939    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647943    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647947    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647953    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647958    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647963    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647970    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:29.650245 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647976    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647981    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647993    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.647999    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648003    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648008    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648013    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648017    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648022    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648026    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648030    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648034    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648039    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648043    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648047    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648051    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648055    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648060    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648688    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648699    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:29.650979 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648704    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648708    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648714    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648718    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648722    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648727    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648731    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648735    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648740    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648744    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648748    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648752    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648756    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648761    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648766    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648770    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648774    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648778    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648782    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648786    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:29.651556 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648789    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648794    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648799    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648804    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648807    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648811    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648815    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648819    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648823    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648827    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648832    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648837    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648841    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648845    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648850    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648854    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648858    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648862    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648866    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648870    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:29.652148 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648874    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648878    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648882    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648886    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648890    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648894    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648898    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648904    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648908    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648912    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648915    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648920    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648924    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648928    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648932    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648936    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648940    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648947    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648952    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:29.652900 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648957    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648962    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648967    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648972    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648977    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648982    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648986    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648990    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648994    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.648998    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649002    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649007    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649011    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649015    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649019    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649025    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649033    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649038    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649043    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:29.653363 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649048    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649052    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649057    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649061    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649065    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649069    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649073    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649077    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649081    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649085    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649089    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649094    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649098    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649102    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649106    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649110    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649114    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649120    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.649125    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649229    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 09:56:29.653971 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649239    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649250    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649257    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649263    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649268    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649288    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649295    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649300    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649305    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649310    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649315    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649320    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649324    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649329    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649333    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649339    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649344    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649350    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649355    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649360    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649364    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649370    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649376    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 09:56:29.654455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649381    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649386    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649391    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649396    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649401    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649406    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649412    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649417    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649423    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649429    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649433    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649438    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649442    2577 flags.go:64] FLAG: --enable-server="true"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649447    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649455    2577 flags.go:64] FLAG: --event-burst="100"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649460    2577 flags.go:64] FLAG: --event-qps="50"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649465    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649470    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649474    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649481    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649485    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649490    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649494    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649499    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649504    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 09:56:29.655022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649509    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649514    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649519    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649523    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649545    2577 flags.go:64] FLAG: --feature-gates=""
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649550    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649555    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649559    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649564    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649569    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649574    2577 flags.go:64] FLAG: --help="false"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649580    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-137-140.ec2.internal"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649585    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649589    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649594    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649600    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649605    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649610    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649615    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649619    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649624    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649628    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649633    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649638    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 20 09:56:29.655676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649642    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649647    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649652    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649656    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649663    2577 flags.go:64] FLAG: --lock-file=""
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649668    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649673    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649677    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649686    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649692    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649697    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649701    2577 flags.go:64] FLAG: --logging-format="text"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649706    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649711    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649716    2577 flags.go:64] FLAG: --manifest-url=""
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649721    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649729    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649734    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649741    2577 flags.go:64] FLAG: --max-pods="110"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649745    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649750    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649754    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649759    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649764    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649768    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 09:56:29.656258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649773    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649784    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649789    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649793    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649798    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649803    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649812    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649816    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649821    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649825    2577 flags.go:64] FLAG: --port="10250"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649830    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649837    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04fdb6b94496ef785"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649842    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649846    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649851    2577 flags.go:64] FLAG: --register-node="true"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649856    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649861    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649867    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649872    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649876    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649882    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649887    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649892    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649897    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649903    2577 flags.go:64] FLAG: --runonce="false"
Jul 20 09:56:29.656898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649907    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649912    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649917    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649921    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649926    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649930    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649935    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649940    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649944    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649948    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649953    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649957    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649962    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649967    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649971    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649980    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649984    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.649988    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650441    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650447    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650452    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650457    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650462    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650467    2577 flags.go:64] FLAG: --v="2"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650474    2577 flags.go:64] FLAG: --version="false"
Jul 20 09:56:29.657598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650482    2577 flags.go:64] FLAG: --vmodule=""
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650489    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.650494    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650699    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650707    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650712    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650718    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650723    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650727    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650731    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650737    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650741    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650745    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650749    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650753    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650757    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650761    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650765    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650769    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650773    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:29.658211 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650777    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650781    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650787    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650794    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650798    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650804    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650808    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650812    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650816    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650820    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650825    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650829    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650833    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650838    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650842    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650846    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650850    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650853    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650858    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:29.658738 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650862    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650866    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650870    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650874    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650878    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650882    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650885    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650890    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650894    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650897    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650901    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650906    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650910    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650914    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650918    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650922    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650926    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650930    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650935    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650939    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:29.659187 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650943    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650948    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650952    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650956    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650960    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650964    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650968    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650973    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650977    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650983    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650988    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650992    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.650997    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651002    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651007    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651012    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651016    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651021    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:29.659792 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651025    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651029    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651033    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651037    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651041    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651045    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651048    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651052    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651056    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651060    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651064    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651068    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651074    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651077    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651081    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651086    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651089    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651093    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651098    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651102    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:29.660220 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651106    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651111    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651116    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651120    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.651124    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.651679    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.658718    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.658735    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.659391    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.659399    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659456    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659461    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659464    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659467    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659470    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:29.660745 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659473    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659475    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659478    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659480    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659483    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659486    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659488    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659491    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659494    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659497    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659500    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659502    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659505    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659507    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659510    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659512    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659514    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659517    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659519    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:29.661125 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659548    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659552    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659554    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659557    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659559    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659562    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659567    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659570    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659573    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659575    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659578    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659580    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659583    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659585    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659588    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659590    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659593    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659595    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659598    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659600    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:29.661592 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659602    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659605    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659608    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659610    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659613    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659617    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659620    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659623    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659625    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659628    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659631    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659634    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659636    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659639    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659642    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659644    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659647    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659650    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659652    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:29.662063 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659655    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659657    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659659    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659662    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659664    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659666    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659669    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659671    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659675    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659678    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659681    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659683    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659686    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659689    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659692    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659695    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659697    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659700    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659702    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:29.662546 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659705    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659707    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659710    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659712    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659715    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659718    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659720    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659723    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659726    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659728    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659730    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659733    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659736    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659738    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659741    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659743    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:29.663023 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659745    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.659750    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659883    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659888    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659891    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659894    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659896    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659899    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659902    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659904    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659907    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659909    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659912    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659914    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659917    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:29.663423 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659919    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659922    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659924    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659927    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659929    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659932    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659935    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659938    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659940    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659942    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659947    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659950    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659953    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659955    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659958    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659961    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659963    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659966    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659968    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:29.663864 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659970    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659973    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659976    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659979    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659981    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659983    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659986    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659988    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659990    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659993    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659995    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.659998    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660000    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660002    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660005    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660007    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660010    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660012    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660015    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660017    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:29.664330 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660020    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660022    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660024    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660026    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660029    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660032    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660034    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660037    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660039    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660042    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660044    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660046    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660049    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660051    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660054    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660056    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660058    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660061    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660063    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660065    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:29.664842 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660068    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660070    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660072    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660075    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660077    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660079    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660082    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660085    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660087    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660089    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660092    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660094    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660096    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660099    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660101    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660104    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660106    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660109    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660113    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:29.665352 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660116    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660119    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660121    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660124    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660127    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660130    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660133    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:29.660135    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.660139    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.660250    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.662712    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.664007    2577 server.go:1019] "Starting client certificate rotation"
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.664109    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:29.665835 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.664148    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:29.666416 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.666402    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:29.667556 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.667515    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:29.675126 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.675108    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 20 09:56:29.676928 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.676910    2577 log.go:25] "Validated CRI v1 image API"
Jul 20 09:56:29.680115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.680097    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 09:56:29.682401 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.682382    2577 fs.go:135] Filesystem UUIDs: map[13c5d73f-65d6-458c-97b8-ac3e794c4412:/dev/nvme0n1p3 1c1bc4b3-25d7-42a8-963e-cea194e48935:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 20 09:56:29.682463 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.682401    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 09:56:29.687559 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.687431    2577 manager.go:216] Machine: {Timestamp:2026-07-20 09:56:29.686034566 +0000 UTC m=+0.318300624 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100104 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b4f9f0b821822494d11a02d704b8 SystemUUID:ec24b4f9-f0b8-2182-2494-d11a02d704b8 BootID:286b476e-b25a-4afa-8b11-7883d228aed7 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fa:2f:92:8d:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fa:2f:92:8d:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:dd:3d:35:96:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 09:56:29.687559 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.687553    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 09:56:29.687692 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.687632    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 09:56:29.690877 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.690853    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 09:56:29.691043 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.690882    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 09:56:29.691681 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.691669    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 09:56:29.691733 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.691687    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 09:56:29.691733 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.691698    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:29.692541 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.692515    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:29.692694 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.692677    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:56:29.693687 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.693674    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 09:56:29.696825 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.696811    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:29.696957 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.696946    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 09:56:29.697011 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.696984    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 09:56:29.697011 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.696997    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 09:56:29.697102 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.697020    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 09:56:29.697102 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.697034    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 20 09:56:29.697102 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.697046    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 09:56:29.697435 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.697421    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:29.697435 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.697446    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:29.699899 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.699886    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 09:56:29.701351 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.701338    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 09:56:29.701395 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.701369    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 09:56:29.701970 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.701951    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 09:56:29.701970 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.701969    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 09:56:29.701970 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.701975    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.701981    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.701996    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702002    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702011    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702016    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702023    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702028    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702041    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 09:56:29.702110 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702047    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 09:56:29.702786 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702775    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 09:56:29.702786 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.702785    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 09:56:29.706249 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.706237    2577 server.go:1268] "Started kubelet"
Jul 20 09:56:29.706363 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.706310    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 09:56:29.706508 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.706320    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 09:56:29.706616 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.706548    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 09:56:29.710607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.710054    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 09:56:29.710607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.710414    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 09:56:29.710607 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.710565    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 09:56:29.710309 ip-10-0-137-140 systemd[1]: Started Kubernetes Kubelet.
Jul 20 09:56:29.710997 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.710664    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 09:56:29.711983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.711947    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 09:56:29.712762 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.712495    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 09:56:29.715048 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.713816    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f0646bacf4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.706210548 +0000 UTC m=+0.338476587,LastTimestamp:2026-07-20 09:56:29.706210548 +0000 UTC m=+0.338476587,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.719140 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.719125    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 09:56:29.719230 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.719136    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 09:56:29.719315 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.719289    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:29.719967 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.719803    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 09:56:29.719967 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.719820    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 09:56:29.719967 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.719965    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 09:56:29.720108 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.719974    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 09:56:29.720234 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720208    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 09:56:29.720708 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720692    2577 factory.go:153] Registering CRI-O factory
Jul 20 09:56:29.720795 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720736    2577 factory.go:223] Registration of the crio container factory successfully
Jul 20 09:56:29.720795 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720788    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 09:56:29.720888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720798    2577 factory.go:55] Registering systemd factory
Jul 20 09:56:29.720888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720806    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 20 09:56:29.720888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720827    2577 factory.go:103] Registering Raw factory
Jul 20 09:56:29.720888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.720841    2577 manager.go:1196] Started watching for new ooms in manager
Jul 20 09:56:29.721313 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.721295    2577 manager.go:349] Starting recovery of all containers
Jul 20 09:56:29.721703 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.721670    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 20 09:56:29.721893 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.721869    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 20 09:56:29.732742 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.732723    2577 manager.go:354] Recovery completed
Jul 20 09:56:29.736917 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.736903    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:29.738864 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.738841    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:29.738932 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.738876    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:29.738932 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.738888    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:29.739383 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.739369    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 09:56:29.739383 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.739380    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 09:56:29.739490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.739394    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:29.739490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.739393    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:29.740593 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.740579    2577 policy_none.go:49] "None policy: Start"
Jul 20 09:56:29.740669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.740596    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 09:56:29.740669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.740608    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 09:56:29.740669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.740618    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:29.740669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.740645    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:29.740669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.740655    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:29.741557 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.741546    2577 policy_none.go:47] "Start"
Jul 20 09:56:29.754842 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.754632    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f0665dd706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-140.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.738858246 +0000 UTC m=+0.371124290,LastTimestamp:2026-07-20 09:56:29.738858246 +0000 UTC m=+0.371124290,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.763030 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.762731    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f0665e317f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-140.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.738881407 +0000 UTC m=+0.371147449,LastTimestamp:2026-07-20 09:56:29.738881407 +0000 UTC m=+0.371147449,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.765505 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.765172    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f0665e59b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-140.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.738891703 +0000 UTC m=+0.371157745,LastTimestamp:2026-07-20 09:56:29.738891703 +0000 UTC m=+0.371157745,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.773009 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.772940    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-140.ec2.internal.18c3f6f0665dd706\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f0665dd706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-140.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.738858246 +0000 UTC m=+0.371124290,LastTimestamp:2026-07-20 09:56:29.740630702 +0000 UTC m=+0.372896741,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.785629    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-140.ec2.internal.18c3f6f0665e317f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f0665e317f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-140.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.738881407 +0000 UTC m=+0.371147449,LastTimestamp:2026-07-20 09:56:29.740650083 +0000 UTC m=+0.372916122,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.787123    2577 manager.go:337] "Starting Device Plugin manager"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.787159    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.787173    2577 server.go:87] "Starting device plugin registration server"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.787445    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.787459    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.787569    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.787645    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.787654    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.788192    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.788182    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.788234    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.789575    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.789611    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.789633    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 09:56:29.790236 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.789662    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 20 09:56:29.796120 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.796052    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-140.ec2.internal.18c3f6f0665e59b7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f0665e59b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-140.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.738891703 +0000 UTC m=+0.371157745,LastTimestamp:2026-07-20 09:56:29.740660063 +0000 UTC m=+0.372926101,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.796120 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.796112    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jul 20 09:56:29.806734 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.806663    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-140.ec2.internal.18c3f6f069644b31  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-140.ec2.internal,UID:ip-10-0-137-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-140.ec2.internal,},FirstTimestamp:2026-07-20 09:56:29.789612849 +0000 UTC m=+0.421878874,LastTimestamp:2026-07-20 09:56:29.789612849 +0000 UTC m=+0.421878874,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-140.ec2.internal,}"
Jul 20 09:56:29.816169 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.816152    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wmk7"
Jul 20 09:56:29.824353 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.824338    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wmk7"
Jul 20 09:56:29.887902 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.887845    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:29.889146 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.889130    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:29.889220 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.889163    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:29.889220 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.889177    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:29.889220 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.889201    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-140.ec2.internal"
Jul 20 09:56:29.889709 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.889691    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"]
Jul 20 09:56:29.889787 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.889749    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:29.890500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.890486    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:29.890579 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.890511    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:29.890579 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.890521    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:29.891928 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.891916    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:29.892056 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:29.892109 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892076    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:29.892640 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892623    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:29.892712 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892651    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:29.892712 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892666    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:29.892712 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892699    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:29.892807 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892717    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:29.892807 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.892730    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:29.893895 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.893881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:29.893971 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.893909    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:29.894613 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.894598    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:29.894679 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.894646    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:29.894679 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.894656    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:29.905830 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:29.905814    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-140.ec2.internal"
Jul 20 09:56:29.905928 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.905838    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-140.ec2.internal\": node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:29.911077 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.911056    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-140.ec2.internal\" not found" node="ip-10-0-137-140.ec2.internal"
Jul 20 09:56:29.915633 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.915615    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-140.ec2.internal\" not found" node="ip-10-0-137-140.ec2.internal"
Jul 20 09:56:29.925241 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:29.925218    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.020469 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.020439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1034d206c86d836b9e3ccaac371542ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal\" (UID: \"1034d206c86d836b9e3ccaac371542ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.020469 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.020470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1034d206c86d836b9e3ccaac371542ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal\" (UID: \"1034d206c86d836b9e3ccaac371542ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.020670 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.020491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0439c32b7ecbcfae8e78a0f3767df0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-140.ec2.internal\" (UID: \"5b0439c32b7ecbcfae8e78a0f3767df0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.025566 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.025537    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.120986 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.120934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1034d206c86d836b9e3ccaac371542ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal\" (UID: \"1034d206c86d836b9e3ccaac371542ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.120986 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.120987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0439c32b7ecbcfae8e78a0f3767df0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-140.ec2.internal\" (UID: \"5b0439c32b7ecbcfae8e78a0f3767df0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.121212 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.121012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1034d206c86d836b9e3ccaac371542ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal\" (UID: \"1034d206c86d836b9e3ccaac371542ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.121212 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.121050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1034d206c86d836b9e3ccaac371542ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal\" (UID: \"1034d206c86d836b9e3ccaac371542ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.121212 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.121071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b0439c32b7ecbcfae8e78a0f3767df0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-140.ec2.internal\" (UID: \"5b0439c32b7ecbcfae8e78a0f3767df0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.121212 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.121059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1034d206c86d836b9e3ccaac371542ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal\" (UID: \"1034d206c86d836b9e3ccaac371542ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.126035 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.126012    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.213568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.213510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.221371 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.221348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.226568 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.226546    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.327143 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.327103    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.427612 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.427579    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.528194 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.528134    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.628676 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.628643    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.663961 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.663933    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 09:56:30.729221 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.729192    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.777126 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:30.777092    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1034d206c86d836b9e3ccaac371542ce.slice/crio-b1b3619b86ddaefe581d216e706d65670766a6af756d5c789829827991180da0 WatchSource:0}: Error finding container b1b3619b86ddaefe581d216e706d65670766a6af756d5c789829827991180da0: Status 404 returned error can't find the container with id b1b3619b86ddaefe581d216e706d65670766a6af756d5c789829827991180da0
Jul 20 09:56:30.777601 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:30.777578    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0439c32b7ecbcfae8e78a0f3767df0.slice/crio-ba541aadbbeb96f2eb6e8657ea5bf768f24085afdaa3b0c46779877c26ac8aeb WatchSource:0}: Error finding container ba541aadbbeb96f2eb6e8657ea5bf768f24085afdaa3b0c46779877c26ac8aeb: Status 404 returned error can't find the container with id ba541aadbbeb96f2eb6e8657ea5bf768f24085afdaa3b0c46779877c26ac8aeb
Jul 20 09:56:30.782055 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.782038    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 09:56:30.794955 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.793903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal" event={"ID":"1034d206c86d836b9e3ccaac371542ce","Type":"ContainerStarted","Data":"b1b3619b86ddaefe581d216e706d65670766a6af756d5c789829827991180da0"}
Jul 20 09:56:30.796196 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.796174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal" event={"ID":"5b0439c32b7ecbcfae8e78a0f3767df0","Type":"ContainerStarted","Data":"ba541aadbbeb96f2eb6e8657ea5bf768f24085afdaa3b0c46779877c26ac8aeb"}
Jul 20 09:56:30.828603 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.828565    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 09:51:29 +0000 UTC" deadline="2028-02-07 09:53:39.828192704 +0000 UTC"
Jul 20 09:56:30.828603 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.828598    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13607h57m8.999597747s"
Jul 20 09:56:30.829654 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:30.829637    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-140.ec2.internal\" not found"
Jul 20 09:56:30.865657 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.865639    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:30.920195 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.920153    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.933086 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.933060    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:56:30.934846 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.934833    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"
Jul 20 09:56:30.943517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:30.943497    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:56:31.166055 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.165950    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:31.166252 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.166237    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:31.669613 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.669584    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:31.697580 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.697482    2577 apiserver.go:52] "Watching apiserver"
Jul 20 09:56:31.703562 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.703534    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 09:56:31.704962 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.704929    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-q5ws5","openshift-image-registry/node-ca-x48x8","openshift-multus/multus-w49lz","openshift-multus/network-metrics-daemon-mg256","openshift-ovn-kubernetes/ovnkube-node-8h6hb","kube-system/global-pull-secret-syncer-tjzjf","kube-system/konnectivity-agent-dgsrm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp","openshift-dns/node-resolver-2wmz4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal","openshift-multus/multus-additional-cni-plugins-qbc85","openshift-network-diagnostics/network-check-target-7lgzb","openshift-network-operator/iptables-alerter-hv4xk","kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal"]
Jul 20 09:56:31.706669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.706638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.709046 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.708616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.710612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.709418    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 09:56:31.710612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.709479    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 09:56:31.710612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.709751    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 09:56:31.710612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.709948    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hns69\""
Jul 20 09:56:31.710612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.710108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 09:56:31.710612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.710351    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 09:56:31.710983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.710634    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 09:56:31.710983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.710706    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 09:56:31.710983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.710707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwbvg\""
Jul 20 09:56:31.710983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.710738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:31.710983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.710979    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 09:56:31.711912 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.711891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:31.712053 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.711966    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:31.713449 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.713429    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8n5sn\""
Jul 20 09:56:31.713688 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.713667    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 09:56:31.713954 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.713931    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 09:56:31.714182 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.714165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.715900 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.715875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.716103 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.716072    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:31.716345 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.716326    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 09:56:31.717730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.717066    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 09:56:31.717730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.717116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.717730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.717176    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zbzb8\""
Jul 20 09:56:31.717730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.717577    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 09:56:31.719342 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.719321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.719439 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.719382    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 09:56:31.719683 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.719664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 09:56:31.719978 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.719957    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 09:56:31.720056 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.720023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnhbr\""
Jul 20 09:56:31.720234 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.720215    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 09:56:31.720286 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.720250    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:56:31.720286 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.720275    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 09:56:31.722064 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.722033    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 09:56:31.722162 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.722121    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2k2zq\""
Jul 20 09:56:31.722408 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.722356    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 09:56:31.722408 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.722373    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 09:56:31.722408 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.722390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.723640 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.723619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.724368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.724349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z5h7s\""
Jul 20 09:56:31.724586 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.724566    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 09:56:31.724690 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.724625    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 09:56:31.725183 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.725153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:31.725277 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.725224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:31.725538 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.725520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6vlz\""
Jul 20 09:56:31.725885 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.725866    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 09:56:31.726799 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.726383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.728439 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728422    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 09:56:31.728614 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728586    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 09:56:31.728706 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728671    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:56:31.728759 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-slash\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.728839 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-var-lib-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.728903 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.728903 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-cni-netd\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729007 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c-agent-certs\") pod \"konnectivity-agent-dgsrm\" (UID: \"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c\") " pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:31.729007 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:31.729007 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.728996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-device-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.729138 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-sys-fs\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.729138 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-kubelet\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729138 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.729138 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.729285 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-etc-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729285 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-ovn\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729285 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b500adc8-0666-4363-9441-4dea722c9727-tmp\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729285 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a99480-abae-43bb-86a7-47ef8e6836d2-serviceca\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.729285 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acf5741f-e17e-4366-bd46-a3ec4dd047ce-dbus\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysctl-conf\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-os-release\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgs6r\" (UniqueName: \"kubernetes.io/projected/dec8fc27-3040-4b3f-91ea-b6696b3aae15-kube-api-access-qgs6r\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a99480-abae-43bb-86a7-47ef8e6836d2-host\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acf5741f-e17e-4366-bd46-a3ec4dd047ce-kubelet-config\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-system-cni-dir\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c-konnectivity-ca\") pod \"konnectivity-agent-dgsrm\" (UID: \"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c\") " pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:31.729517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-log-socket\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-ovnkube-config\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-env-overrides\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcv8\" (UniqueName: \"kubernetes.io/projected/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-kube-api-access-zqcv8\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-kubernetes\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729677    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9vg2s\""
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-systemd\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-host\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4d4b\" (UniqueName: \"kubernetes.io/projected/b500adc8-0666-4363-9441-4dea722c9727-kube-api-access-l4d4b\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwn6\" (UniqueName: \"kubernetes.io/projected/8c74b748-ca84-4956-a79b-c56f113ae50a-kube-api-access-rzwn6\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-sys\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-var-lib-kubelet\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.729913 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29c3298f-7d97-47d3-a41a-b064e8935ff1-hosts-file\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-run\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cnibin\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.729988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-socket-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-lib-modules\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c3298f-7d97-47d3-a41a-b064e8935ff1-tmp-dir\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-node-log\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-cni-bin\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-ovnkube-script-lib\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cqgn\" (UniqueName: \"kubernetes.io/projected/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-kube-api-access-7cqgn\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-modprobe-d\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b500adc8-0666-4363-9441-4dea722c9727-etc-tuned\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdvn\" (UniqueName: \"kubernetes.io/projected/29c3298f-7d97-47d3-a41a-b064e8935ff1-kube-api-access-7wdvn\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt5j\" (UniqueName: \"kubernetes.io/projected/e0a99480-abae-43bb-86a7-47ef8e6836d2-kube-api-access-tpt5j\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.730707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-run-netns\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.731396 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-systemd\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.731396 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c74b748-ca84-4956-a79b-c56f113ae50a-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.731396 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-systemd-units\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.731396 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-registration-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.731396 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysctl-d\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.731396 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysconfig\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.731396 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.730742    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:56:31.752738 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.752702    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lkh56"
Jul 20 09:56:31.759957 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.759643    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lkh56"
Jul 20 09:56:31.822472 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.822436    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 09:56:31.831360 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-netns\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.831500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-device-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.831500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-sys-fs\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.831500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd4sc\" (UniqueName: \"kubernetes.io/projected/1a837a28-6f13-4317-abc9-a1601de1f28f-kube-api-access-zd4sc\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.831500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-kubelet\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.831500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-device-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.831500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-sys-fs\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.831500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-kubelet\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-system-cni-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-socket-dir-parent\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a837a28-6f13-4317-abc9-a1601de1f28f-host-slash\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.831682    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-etc-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-ovn\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.831750    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret podName:acf5741f-e17e-4366-bd46-a3ec4dd047ce nodeName:}" failed. No retries permitted until 2026-07-20 09:56:32.331729921 +0000 UTC m=+2.963995978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret") pod "global-pull-secret-syncer-tjzjf" (UID: "acf5741f-e17e-4366-bd46-a3ec4dd047ce") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-etc-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-ovn\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b500adc8-0666-4363-9441-4dea722c9727-tmp\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a99480-abae-43bb-86a7-47ef8e6836d2-serviceca\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acf5741f-e17e-4366-bd46-a3ec4dd047ce-dbus\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.831906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysctl-conf\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-cni-multus\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hb5t\" (UniqueName: \"kubernetes.io/projected/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-kube-api-access-7hb5t\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.831978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysctl-conf\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-os-release\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgs6r\" (UniqueName: \"kubernetes.io/projected/dec8fc27-3040-4b3f-91ea-b6696b3aae15-kube-api-access-qgs6r\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832048    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a99480-abae-43bb-86a7-47ef8e6836d2-host\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-os-release\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acf5741f-e17e-4366-bd46-a3ec4dd047ce-dbus\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acf5741f-e17e-4366-bd46-a3ec4dd047ce-kubelet-config\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a99480-abae-43bb-86a7-47ef8e6836d2-host\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-cni-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-system-cni-dir\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acf5741f-e17e-4366-bd46-a3ec4dd047ce-kubelet-config\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c-konnectivity-ca\") pod \"konnectivity-agent-dgsrm\" (UID: \"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c\") " pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:31.832721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-log-socket\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-system-cni-dir\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-ovnkube-config\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-env-overrides\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-log-socket\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a99480-abae-43bb-86a7-47ef8e6836d2-serviceca\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcv8\" (UniqueName: \"kubernetes.io/projected/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-kube-api-access-zqcv8\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-kubernetes\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-systemd\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-host\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4d4b\" (UniqueName: \"kubernetes.io/projected/b500adc8-0666-4363-9441-4dea722c9727-kube-api-access-l4d4b\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.833552 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-kubernetes\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-cni-binary-copy\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwn6\" (UniqueName: \"kubernetes.io/projected/8c74b748-ca84-4956-a79b-c56f113ae50a-kube-api-access-rzwn6\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-sys\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-var-lib-kubelet\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29c3298f-7d97-47d3-a41a-b064e8935ff1-hosts-file\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-os-release\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c-konnectivity-ca\") pod \"konnectivity-agent-dgsrm\" (UID: \"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c\") " pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29c3298f-7d97-47d3-a41a-b064e8935ff1-hosts-file\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-var-lib-kubelet\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-sys\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-cni-bin\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-run\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-env-overrides\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cnibin\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.834390 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.832999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-socket-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-lib-modules\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-ovnkube-config\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c3298f-7d97-47d3-a41a-b064e8935ff1-tmp-dir\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-systemd\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-cnibin\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-hostroot\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-socket-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-run\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-host\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cnibin\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-node-log\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-lib-modules\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.835428 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-cni-bin\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-node-log\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29c3298f-7d97-47d3-a41a-b064e8935ff1-tmp-dir\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-cni-bin\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-ovnkube-script-lib\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cqgn\" (UniqueName: \"kubernetes.io/projected/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-kube-api-access-7cqgn\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-modprobe-d\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b500adc8-0666-4363-9441-4dea722c9727-etc-tuned\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdvn\" (UniqueName: \"kubernetes.io/projected/29c3298f-7d97-47d3-a41a-b064e8935ff1-kube-api-access-7wdvn\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-modprobe-d\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt5j\" (UniqueName: \"kubernetes.io/projected/e0a99480-abae-43bb-86a7-47ef8e6836d2-kube-api-access-tpt5j\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dec8fc27-3040-4b3f-91ea-b6696b3aae15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-run-netns\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-systemd\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c74b748-ca84-4956-a79b-c56f113ae50a-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c74b748-ca84-4956-a79b-c56f113ae50a-ovnkube-script-lib\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-run-netns\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.836357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-daemon-config\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-multus-certs\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-systemd\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-systemd-units\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.833995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-registration-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysctl-d\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-conf-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-systemd-units\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a837a28-6f13-4317-abc9-a1601de1f28f-iptables-alerter-script\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-k8s-cni-cncf-io\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-registration-dir\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysconfig\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-kubelet\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysconfig\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b500adc8-0666-4363-9441-4dea722c9727-etc-sysctl-d\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-etc-kubernetes\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:31.837036 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-slash\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-var-lib-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-slash\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-cni-netd\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-run-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c-agent-certs\") pod \"konnectivity-agent-dgsrm\" (UID: \"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c\") " pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-host-cni-netd\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.834284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c74b748-ca84-4956-a79b-c56f113ae50a-var-lib-openvswitch\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.834419    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.834455    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:32.33444253 +0000 UTC m=+2.966708556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.835644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b500adc8-0666-4363-9441-4dea722c9727-tmp\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.835950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b500adc8-0666-4363-9441-4dea722c9727-etc-tuned\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.836660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c74b748-ca84-4956-a79b-c56f113ae50a-ovn-node-metrics-cert\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.837724 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.837119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c-agent-certs\") pod \"konnectivity-agent-dgsrm\" (UID: \"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c\") " pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:31.841120 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.841063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgs6r\" (UniqueName: \"kubernetes.io/projected/dec8fc27-3040-4b3f-91ea-b6696b3aae15-kube-api-access-qgs6r\") pod \"multus-additional-cni-plugins-qbc85\" (UID: \"dec8fc27-3040-4b3f-91ea-b6696b3aae15\") " pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:31.841861 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.841836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcv8\" (UniqueName: \"kubernetes.io/projected/33e41221-b006-4b07-a7cd-2dc50ebe0eb9-kube-api-access-zqcv8\") pod \"aws-ebs-csi-driver-node-lmftp\" (UID: \"33e41221-b006-4b07-a7cd-2dc50ebe0eb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:31.843314 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.843294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt5j\" (UniqueName: \"kubernetes.io/projected/e0a99480-abae-43bb-86a7-47ef8e6836d2-kube-api-access-tpt5j\") pod \"node-ca-x48x8\" (UID: \"e0a99480-abae-43bb-86a7-47ef8e6836d2\") " pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:31.844111 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.844085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdvn\" (UniqueName: \"kubernetes.io/projected/29c3298f-7d97-47d3-a41a-b064e8935ff1-kube-api-access-7wdvn\") pod \"node-resolver-2wmz4\" (UID: \"29c3298f-7d97-47d3-a41a-b064e8935ff1\") " pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:31.844567 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.844538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4d4b\" (UniqueName: \"kubernetes.io/projected/b500adc8-0666-4363-9441-4dea722c9727-kube-api-access-l4d4b\") pod \"tuned-q5ws5\" (UID: \"b500adc8-0666-4363-9441-4dea722c9727\") " pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:31.844700 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.844675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cqgn\" (UniqueName: \"kubernetes.io/projected/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-kube-api-access-7cqgn\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:31.845476 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.845448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwn6\" (UniqueName: \"kubernetes.io/projected/8c74b748-ca84-4956-a79b-c56f113ae50a-kube-api-access-rzwn6\") pod \"ovnkube-node-8h6hb\" (UID: \"8c74b748-ca84-4956-a79b-c56f113ae50a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:31.935089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-conf-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a837a28-6f13-4317-abc9-a1601de1f28f-iptables-alerter-script\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-k8s-cni-cncf-io\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-kubelet\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-k8s-cni-cncf-io\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-conf-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-etc-kubernetes\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-etc-kubernetes\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-kubelet\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-netns\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-netns\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd4sc\" (UniqueName: \"kubernetes.io/projected/1a837a28-6f13-4317-abc9-a1601de1f28f-kube-api-access-zd4sc\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-system-cni-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-socket-dir-parent\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a837a28-6f13-4317-abc9-a1601de1f28f-host-slash\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-system-cni-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-cni-multus\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hb5t\" (UniqueName: \"kubernetes.io/projected/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-kube-api-access-7hb5t\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-socket-dir-parent\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-cni-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a837a28-6f13-4317-abc9-a1601de1f28f-host-slash\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-cni-multus\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-cni-binary-copy\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-os-release\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-cni-bin\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a837a28-6f13-4317-abc9-a1601de1f28f-iptables-alerter-script\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-cni-dir\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.935832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-var-lib-cni-bin\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-os-release\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-cnibin\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-hostroot\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-hostroot\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-daemon-config\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-cnibin\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-multus-certs\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.935979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-host-run-multus-certs\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.936145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-cni-binary-copy\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.936629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.936395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-multus-daemon-config\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.943327 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.943301    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:31.943327 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.943331    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:31.943707 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.943346    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z9jxn for pod openshift-network-diagnostics/network-check-target-7lgzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:31.943707 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:31.943408    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn podName:62eb33ef-e544-4ab4-aa09-542e5586e957 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:32.443391065 +0000 UTC m=+3.075657112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z9jxn" (UniqueName: "kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn") pod "network-check-target-7lgzb" (UID: "62eb33ef-e544-4ab4-aa09-542e5586e957") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:31.945707 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.945686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hb5t\" (UniqueName: \"kubernetes.io/projected/bb539c41-358c-4ed9-8ae4-f6b4e43d35f0-kube-api-access-7hb5t\") pod \"multus-w49lz\" (UID: \"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0\") " pod="openshift-multus/multus-w49lz"
Jul 20 09:56:31.945802 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:31.945693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd4sc\" (UniqueName: \"kubernetes.io/projected/1a837a28-6f13-4317-abc9-a1601de1f28f-kube-api-access-zd4sc\") pod \"iptables-alerter-hv4xk\" (UID: \"1a837a28-6f13-4317-abc9-a1601de1f28f\") " pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:32.023159 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.023079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbc85"
Jul 20 09:56:32.031734 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.031707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x48x8"
Jul 20 09:56:32.042992 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.042964    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w49lz"
Jul 20 09:56:32.048088 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.048068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:32.055191 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.055168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:32.064186 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.064165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q5ws5"
Jul 20 09:56:32.070847 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.070826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp"
Jul 20 09:56:32.077839 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.077820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wmz4"
Jul 20 09:56:32.085612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.085587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hv4xk"
Jul 20 09:56:32.339630 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.339521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:32.339807 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.339649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:32.339807 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.339682    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:32.339807 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.339762    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret podName:acf5741f-e17e-4366-bd46-a3ec4dd047ce nodeName:}" failed. No retries permitted until 2026-07-20 09:56:33.339745994 +0000 UTC m=+3.972012025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret") pod "global-pull-secret-syncer-tjzjf" (UID: "acf5741f-e17e-4366-bd46-a3ec4dd047ce") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:32.339807 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.339762    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:32.339969 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.339819    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:33.339801616 +0000 UTC m=+3.972067661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:32.542019 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.541974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:32.542188 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.542158    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:32.542188 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.542186    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:32.542275 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.542199    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z9jxn for pod openshift-network-diagnostics/network-check-target-7lgzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:32.542307 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.542274    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn podName:62eb33ef-e544-4ab4-aa09-542e5586e957 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:33.542255248 +0000 UTC m=+4.174521281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9jxn" (UniqueName: "kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn") pod "network-check-target-7lgzb" (UID: "62eb33ef-e544-4ab4-aa09-542e5586e957") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:32.761922 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.761823    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:51:31 +0000 UTC" deadline="2028-01-03 02:08:45.282220224 +0000 UTC"
Jul 20 09:56:32.761922 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.761882    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12760h12m12.520343057s"
Jul 20 09:56:32.787450 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.787419    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fdfc3b_ca3a_4a6f_873d_20de0e9b934c.slice/crio-d5b61ee920a694da8fd0866a7c0df6f4e73b471c9f5de062400ef21124857b5c WatchSource:0}: Error finding container d5b61ee920a694da8fd0866a7c0df6f4e73b471c9f5de062400ef21124857b5c: Status 404 returned error can't find the container with id d5b61ee920a694da8fd0866a7c0df6f4e73b471c9f5de062400ef21124857b5c
Jul 20 09:56:32.788571 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.788520    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb539c41_358c_4ed9_8ae4_f6b4e43d35f0.slice/crio-f5743f84e743d097f72e68f37ea6a52c5762f5007b54bedb1f0752d3787ead0e WatchSource:0}: Error finding container f5743f84e743d097f72e68f37ea6a52c5762f5007b54bedb1f0752d3787ead0e: Status 404 returned error can't find the container with id f5743f84e743d097f72e68f37ea6a52c5762f5007b54bedb1f0752d3787ead0e
Jul 20 09:56:32.789336 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.789266    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a837a28_6f13_4317_abc9_a1601de1f28f.slice/crio-fcde0a3e3cb9053086deed1ec9558236106051f1280f23112080ba9ce4ffd802 WatchSource:0}: Error finding container fcde0a3e3cb9053086deed1ec9558236106051f1280f23112080ba9ce4ffd802: Status 404 returned error can't find the container with id fcde0a3e3cb9053086deed1ec9558236106051f1280f23112080ba9ce4ffd802
Jul 20 09:56:32.789887 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.789832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:32.789982 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:32.789954    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:32.793218 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.793197    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec8fc27_3040_4b3f_91ea_b6696b3aae15.slice/crio-1830bd40581ca750e1aa78a3f03a53a3639e4e3728bf7b9a801b708f247bc5b8 WatchSource:0}: Error finding container 1830bd40581ca750e1aa78a3f03a53a3639e4e3728bf7b9a801b708f247bc5b8: Status 404 returned error can't find the container with id 1830bd40581ca750e1aa78a3f03a53a3639e4e3728bf7b9a801b708f247bc5b8
Jul 20 09:56:32.793726 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.793674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c74b748_ca84_4956_a79b_c56f113ae50a.slice/crio-6ee8aae7d52c521455bc1b513a85797de46107d1f25b3fa189435d20456f3970 WatchSource:0}: Error finding container 6ee8aae7d52c521455bc1b513a85797de46107d1f25b3fa189435d20456f3970: Status 404 returned error can't find the container with id 6ee8aae7d52c521455bc1b513a85797de46107d1f25b3fa189435d20456f3970
Jul 20 09:56:32.794481 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.794450    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a99480_abae_43bb_86a7_47ef8e6836d2.slice/crio-0ef0e589097d9c268a59b7cebcb58b02f515ace134f6fbdf36e992ebdfe29f87 WatchSource:0}: Error finding container 0ef0e589097d9c268a59b7cebcb58b02f515ace134f6fbdf36e992ebdfe29f87: Status 404 returned error can't find the container with id 0ef0e589097d9c268a59b7cebcb58b02f515ace134f6fbdf36e992ebdfe29f87
Jul 20 09:56:32.795494 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.795414    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e41221_b006_4b07_a7cd_2dc50ebe0eb9.slice/crio-3a2b3973b3983a878a397b9b59f7a254be60fddcdb02d8ef24dba1113713b8f3 WatchSource:0}: Error finding container 3a2b3973b3983a878a397b9b59f7a254be60fddcdb02d8ef24dba1113713b8f3: Status 404 returned error can't find the container with id 3a2b3973b3983a878a397b9b59f7a254be60fddcdb02d8ef24dba1113713b8f3
Jul 20 09:56:32.796921 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.796099    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb500adc8_0666_4363_9441_4dea722c9727.slice/crio-6f0e0a7c7c643ff797ece85a5868fd8461eb578f8a3c5e4107f39b2083f64bd6 WatchSource:0}: Error finding container 6f0e0a7c7c643ff797ece85a5868fd8461eb578f8a3c5e4107f39b2083f64bd6: Status 404 returned error can't find the container with id 6f0e0a7c7c643ff797ece85a5868fd8461eb578f8a3c5e4107f39b2083f64bd6
Jul 20 09:56:32.798510 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:56:32.798468    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c3298f_7d97_47d3_a41a_b064e8935ff1.slice/crio-eb8c708407fbb21acf796d6140b16b89a19e56f591bef3668321b3fd1c516668 WatchSource:0}: Error finding container eb8c708407fbb21acf796d6140b16b89a19e56f591bef3668321b3fd1c516668: Status 404 returned error can't find the container with id eb8c708407fbb21acf796d6140b16b89a19e56f591bef3668321b3fd1c516668
Jul 20 09:56:32.800600 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.800165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q5ws5" event={"ID":"b500adc8-0666-4363-9441-4dea722c9727","Type":"ContainerStarted","Data":"6f0e0a7c7c643ff797ece85a5868fd8461eb578f8a3c5e4107f39b2083f64bd6"}
Jul 20 09:56:32.801834 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.801787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp" event={"ID":"33e41221-b006-4b07-a7cd-2dc50ebe0eb9","Type":"ContainerStarted","Data":"3a2b3973b3983a878a397b9b59f7a254be60fddcdb02d8ef24dba1113713b8f3"}
Jul 20 09:56:32.803335 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.803300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerStarted","Data":"1830bd40581ca750e1aa78a3f03a53a3639e4e3728bf7b9a801b708f247bc5b8"}
Jul 20 09:56:32.804656 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.804630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dgsrm" event={"ID":"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c","Type":"ContainerStarted","Data":"d5b61ee920a694da8fd0866a7c0df6f4e73b471c9f5de062400ef21124857b5c"}
Jul 20 09:56:32.805998 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.805978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"6ee8aae7d52c521455bc1b513a85797de46107d1f25b3fa189435d20456f3970"}
Jul 20 09:56:32.807285 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.807262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x48x8" event={"ID":"e0a99480-abae-43bb-86a7-47ef8e6836d2","Type":"ContainerStarted","Data":"0ef0e589097d9c268a59b7cebcb58b02f515ace134f6fbdf36e992ebdfe29f87"}
Jul 20 09:56:32.808551 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.808509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w49lz" event={"ID":"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0","Type":"ContainerStarted","Data":"f5743f84e743d097f72e68f37ea6a52c5762f5007b54bedb1f0752d3787ead0e"}
Jul 20 09:56:32.810154 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:32.810122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hv4xk" event={"ID":"1a837a28-6f13-4317-abc9-a1601de1f28f","Type":"ContainerStarted","Data":"fcde0a3e3cb9053086deed1ec9558236106051f1280f23112080ba9ce4ffd802"}
Jul 20 09:56:33.348042 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.347958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:33.348042 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.348017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:33.348256 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.348183    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:33.348256 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.348250    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret podName:acf5741f-e17e-4366-bd46-a3ec4dd047ce nodeName:}" failed. No retries permitted until 2026-07-20 09:56:35.348228534 +0000 UTC m=+5.980494583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret") pod "global-pull-secret-syncer-tjzjf" (UID: "acf5741f-e17e-4366-bd46-a3ec4dd047ce") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:33.349732 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.348661    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:33.349732 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.348711    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:35.348696889 +0000 UTC m=+5.980962914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:33.550553 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.549870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:33.550553 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.550089    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:33.550553 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.550111    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:33.550553 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.550125    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z9jxn for pod openshift-network-diagnostics/network-check-target-7lgzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:33.550553 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.550188    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn podName:62eb33ef-e544-4ab4-aa09-542e5586e957 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:35.550168418 +0000 UTC m=+6.182434450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9jxn" (UniqueName: "kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn") pod "network-check-target-7lgzb" (UID: "62eb33ef-e544-4ab4-aa09-542e5586e957") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:33.762174 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.762132    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:51:31 +0000 UTC" deadline="2028-01-21 21:08:12.626162065 +0000 UTC"
Jul 20 09:56:33.762174 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.762173    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13211h11m38.86399377s"
Jul 20 09:56:33.793573 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.792891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:33.793573 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.793024    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:33.793573 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.793417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:33.793573 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:33.793504    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:33.831763 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.831725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wmz4" event={"ID":"29c3298f-7d97-47d3-a41a-b064e8935ff1","Type":"ContainerStarted","Data":"eb8c708407fbb21acf796d6140b16b89a19e56f591bef3668321b3fd1c516668"}
Jul 20 09:56:33.836853 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.835867    2577 generic.go:356] "Generic (PLEG): container finished" podID="1034d206c86d836b9e3ccaac371542ce" containerID="4176687070fce90824e5551353d68f7521404afa31bdc7366a44a398fab5df74" exitCode=0
Jul 20 09:56:33.836853 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.836728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal" event={"ID":"1034d206c86d836b9e3ccaac371542ce","Type":"ContainerDied","Data":"4176687070fce90824e5551353d68f7521404afa31bdc7366a44a398fab5df74"}
Jul 20 09:56:33.845316 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:33.845290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal" event={"ID":"5b0439c32b7ecbcfae8e78a0f3767df0","Type":"ContainerStarted","Data":"ab0d39282c4c3f3693bcebb5d45928335e3ef1f319358375e03a7125d68b91d5"}
Jul 20 09:56:34.790736 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:34.790705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:34.791139 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:34.790814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:34.862631 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:34.862597    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal" event={"ID":"1034d206c86d836b9e3ccaac371542ce","Type":"ContainerStarted","Data":"b52e7fd5d7071658c2199e4d4f084f410ea4d422f882d873bd52660c09224edf"}
Jul 20 09:56:34.886229 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:34.886162    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-140.ec2.internal" podStartSLOduration=3.886141732 podStartE2EDuration="3.886141732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:33.867441617 +0000 UTC m=+4.499707666" watchObservedRunningTime="2026-07-20 09:56:34.886141732 +0000 UTC m=+5.518407781"
Jul 20 09:56:35.365279 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:35.364663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:35.365279 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:35.364719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:35.365279 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.364835    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:35.365279 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.364847    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:35.365279 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.364908    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret podName:acf5741f-e17e-4366-bd46-a3ec4dd047ce nodeName:}" failed. No retries permitted until 2026-07-20 09:56:39.364890212 +0000 UTC m=+9.997156259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret") pod "global-pull-secret-syncer-tjzjf" (UID: "acf5741f-e17e-4366-bd46-a3ec4dd047ce") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:35.365279 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.364929    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:39.364919775 +0000 UTC m=+9.997185805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:35.566298 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:35.566254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:35.566491 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.566470    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:35.566602 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.566494    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:35.566602 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.566507    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z9jxn for pod openshift-network-diagnostics/network-check-target-7lgzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:35.566602 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.566586    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn podName:62eb33ef-e544-4ab4-aa09-542e5586e957 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:39.566567698 +0000 UTC m=+10.198833730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9jxn" (UniqueName: "kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn") pod "network-check-target-7lgzb" (UID: "62eb33ef-e544-4ab4-aa09-542e5586e957") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:35.793471 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:35.792826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:35.793471 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.792946    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:35.793471 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:35.793317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:35.793471 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:35.793414    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:36.790641 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:36.790605    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:36.790896 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:36.790806    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:37.790757 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:37.790653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:37.790757 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:37.790690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:37.791259 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:37.790790    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:37.791259 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:37.790879    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:38.789846 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:38.789807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:38.790028 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:38.789949    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:39.400281 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:39.400032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:39.400281 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:39.400125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:39.400281 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.400192    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:39.400281 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.400223    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:39.400281 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.400266    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret podName:acf5741f-e17e-4366-bd46-a3ec4dd047ce nodeName:}" failed. No retries permitted until 2026-07-20 09:56:47.400243953 +0000 UTC m=+18.032510020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret") pod "global-pull-secret-syncer-tjzjf" (UID: "acf5741f-e17e-4366-bd46-a3ec4dd047ce") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:39.400281 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.400287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:47.40027716 +0000 UTC m=+18.032543191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:39.601603 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:39.601514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:39.601786 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.601710    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:39.601786 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.601741    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:39.601786 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.601754    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z9jxn for pod openshift-network-diagnostics/network-check-target-7lgzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:39.601912 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.601815    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn podName:62eb33ef-e544-4ab4-aa09-542e5586e957 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:47.601797416 +0000 UTC m=+18.234063442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9jxn" (UniqueName: "kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn") pod "network-check-target-7lgzb" (UID: "62eb33ef-e544-4ab4-aa09-542e5586e957") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:39.792193 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:39.792160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:39.792193 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:39.792179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:39.792440 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.792283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:39.794628 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:39.794376    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:40.790454 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:40.790421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:40.790903 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:40.790582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:41.792114 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:41.791631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:41.792114 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:41.791743    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:41.792114 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:41.791807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:41.792114 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:41.791926    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:42.790893 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:42.790858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:42.791088 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:42.790993    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:43.790740 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:43.790707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:43.791114 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:43.790814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:43.791114 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:43.790887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:43.791114 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:43.790963    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:44.790329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:44.790290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:44.790510 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:44.790429    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:45.789896 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:45.789857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:45.790356 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:45.789860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:45.790356 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:45.789979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:45.790356 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:45.790102    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:46.789862 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:46.789829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:46.790111 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:46.789954    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:47.460791 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:47.460756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:47.461020 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:47.460846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:47.461020 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.460924    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:47.461020 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.460940    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:47.461020 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.460988    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.460973047 +0000 UTC m=+34.093239072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:47.461020 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.461001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret podName:acf5741f-e17e-4366-bd46-a3ec4dd047ce nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.46099484 +0000 UTC m=+34.093260866 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret") pod "global-pull-secret-syncer-tjzjf" (UID: "acf5741f-e17e-4366-bd46-a3ec4dd047ce") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:47.661837 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:47.661793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:47.661998 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.661979    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:47.661998 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.661998    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:47.662074 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.662008    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z9jxn for pod openshift-network-diagnostics/network-check-target-7lgzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:47.662074 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.662061    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn podName:62eb33ef-e544-4ab4-aa09-542e5586e957 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:03.662045622 +0000 UTC m=+34.294311669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9jxn" (UniqueName: "kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn") pod "network-check-target-7lgzb" (UID: "62eb33ef-e544-4ab4-aa09-542e5586e957") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:47.790629 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:47.790601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:47.790970 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:47.790607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:47.790970 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.790745    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:47.790970 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:47.790791    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:48.790757 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:48.790717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:48.791212 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:48.790854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:49.790953 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:49.790913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:49.791368 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:49.791038    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:49.791368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:49.791086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:49.791368 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:49.791194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:50.789823 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:50.789792    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:50.789976 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:50.789923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:51.790398 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:51.790362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:51.790828 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:51.790484    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:51.790828 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:51.790367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:51.790828 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:51.790683    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:52.791003 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.790607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:52.791617 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:52.791140    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:52.899171 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.899091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wmz4" event={"ID":"29c3298f-7d97-47d3-a41a-b064e8935ff1","Type":"ContainerStarted","Data":"e298e116756876222dd6d53311ebbc1fd0f0d5b935bd34a7fe561d4fa8073bb5"}
Jul 20 09:56:52.902875 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.902844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"6ba26af2ade11200bb4078ee52854a1aa05840e77c8be91e241c1858c104db29"}
Jul 20 09:56:52.903021 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.902933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"64fc4a645cd8b97827b7582134983f8693e55b6b74224a633b796ab5fda03705"}
Jul 20 09:56:52.903021 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.902986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"c86b865a7abd3100ceb6f6dfbb4ea013ffdad9f7e1a8fd90547530b0cf42acf6"}
Jul 20 09:56:52.903124 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.903033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"a3a3686d171ea1129ad0766b22a78cc1451aefc4632e111a94bc57f544de8e4e"}
Jul 20 09:56:52.903124 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.903081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"7db9d02e693a2cfefae7793a12a43f622297c222f946f37c44b94e81f4694a7c"}
Jul 20 09:56:52.903222 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.903128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"c6383900825e6b3e5d9297c1b5007cd675a5f76bb8b949f77cb59011474594ff"}
Jul 20 09:56:52.904504 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.904473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x48x8" event={"ID":"e0a99480-abae-43bb-86a7-47ef8e6836d2","Type":"ContainerStarted","Data":"b7bdb944631bf2bffb3dbd4520c280d11bf7d5c4b2a46869f106e30131707e4f"}
Jul 20 09:56:52.906037 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.906015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w49lz" event={"ID":"bb539c41-358c-4ed9-8ae4-f6b4e43d35f0","Type":"ContainerStarted","Data":"7c9df47f32059708f55ece7b23f26ebee1224dce2ff85b86d35922fddb52752e"}
Jul 20 09:56:52.907788 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.907762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q5ws5" event={"ID":"b500adc8-0666-4363-9441-4dea722c9727","Type":"ContainerStarted","Data":"d81320e6c74366e24e6d3b7d59965c2f278be13044ef9693623c2ad79b743f16"}
Jul 20 09:56:52.909318 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.909293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp" event={"ID":"33e41221-b006-4b07-a7cd-2dc50ebe0eb9","Type":"ContainerStarted","Data":"5dfaae19c7555d88030eb52d148ce62881d495273163ddcd11d2d39f460c37ca"}
Jul 20 09:56:52.911066 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.911030    2577 generic.go:356] "Generic (PLEG): container finished" podID="dec8fc27-3040-4b3f-91ea-b6696b3aae15" containerID="bd5bfb3ea9b31869c3085624b8ed38c5b32996d39910de7a6bcb5bd2705ffd4f" exitCode=0
Jul 20 09:56:52.911159 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.911113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerDied","Data":"bd5bfb3ea9b31869c3085624b8ed38c5b32996d39910de7a6bcb5bd2705ffd4f"}
Jul 20 09:56:52.913062 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.913039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dgsrm" event={"ID":"a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c","Type":"ContainerStarted","Data":"09a98138cc826befa60e3e919ac63dffa73f24d34ee0425d58a569567d133079"}
Jul 20 09:56:52.914264 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.914231    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-140.ec2.internal" podStartSLOduration=22.914214762 podStartE2EDuration="22.914214762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:34.886508633 +0000 UTC m=+5.518774683" watchObservedRunningTime="2026-07-20 09:56:52.914214762 +0000 UTC m=+23.546480812"
Jul 20 09:56:52.934794 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.934742    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2wmz4" podStartSLOduration=3.736049573 podStartE2EDuration="22.934722851s" totalImagesPullingTime="19.198673278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:52.914738905 +0000 UTC m=+23.547004952" watchObservedRunningTime="2026-07-20 09:56:52.934722851 +0000 UTC m=+23.566988900"
Jul 20 09:56:52.935144 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.935115    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q5ws5" podStartSLOduration=3.741546668 podStartE2EDuration="22.935105494s" totalImagesPullingTime="19.193558826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:52.934583543 +0000 UTC m=+23.566849592" watchObservedRunningTime="2026-07-20 09:56:52.935105494 +0000 UTC m=+23.567371546"
Jul 20 09:56:52.980516 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:52.980465    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w49lz" podStartSLOduration=3.726631553 podStartE2EDuration="22.980447327s" totalImagesPullingTime="19.253815774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:52.979944353 +0000 UTC m=+23.612210400" watchObservedRunningTime="2026-07-20 09:56:52.980447327 +0000 UTC m=+23.612713376"
Jul 20 09:56:53.014766 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.014722    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x48x8" podStartSLOduration=4.814214746 podStartE2EDuration="24.014708356s" totalImagesPullingTime="19.20049361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:52.995244169 +0000 UTC m=+23.627510217" watchObservedRunningTime="2026-07-20 09:56:53.014708356 +0000 UTC m=+23.646974404"
Jul 20 09:56:53.337710 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.337683    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 09:56:53.790953 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.790918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:53.791206 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:53.791028    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:53.791206 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.791090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:53.791561 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:53.791198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:53.796923 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.796826    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T09:56:53.337704001Z","UUID":"5fa68883-401a-4035-9b1b-fdb72343dcc3","Handler":null,"Name":"","Endpoint":""}
Jul 20 09:56:53.798840 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.798804    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 09:56:53.799007 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.798850    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 09:56:53.917335 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.917296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp" event={"ID":"33e41221-b006-4b07-a7cd-2dc50ebe0eb9","Type":"ContainerStarted","Data":"5572470ba416154b611c7c565608df801a93ddae32fbc4c6b8e4f0d18b6981a8"}
Jul 20 09:56:53.918899 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.918871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hv4xk" event={"ID":"1a837a28-6f13-4317-abc9-a1601de1f28f","Type":"ContainerStarted","Data":"731642a6eb03f4f1cf59195831adc038c7413c1840dbdaaa9b124e8e9fb50cd9"}
Jul 20 09:56:53.934521 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:53.934474    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dgsrm" podStartSLOduration=4.733540083 podStartE2EDuration="23.934454426s" totalImagesPullingTime="19.200914343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:53.015448008 +0000 UTC m=+23.647714048" watchObservedRunningTime="2026-07-20 09:56:53.934454426 +0000 UTC m=+24.566720474"
Jul 20 09:56:54.790812 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:54.790780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:54.791103 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:54.790887    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:55.717287 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.717251    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:55.717867 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.717849    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:55.733947 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.733895    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hv4xk" podStartSLOduration=6.533227983 podStartE2EDuration="25.733873997s" totalImagesPullingTime="19.200646014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:53.934024372 +0000 UTC m=+24.566290421" watchObservedRunningTime="2026-07-20 09:56:55.733873997 +0000 UTC m=+26.366140046"
Jul 20 09:56:55.790208 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.790011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:55.790361 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.790009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:55.790361 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:55.790296    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:55.790467 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:55.790393    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:55.927617 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.927074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp" event={"ID":"33e41221-b006-4b07-a7cd-2dc50ebe0eb9","Type":"ContainerStarted","Data":"36b95476fe95846fa98769ea1205229e3a371600ca7ec7c8359228d6115ee10b"}
Jul 20 09:56:55.931001 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.930969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"52ac6e5355eed665e2491ea7fc84affcde9c5484119c4e141f35166cd90b0be9"}
Jul 20 09:56:55.931219 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.931195    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:55.931720 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.931697    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dgsrm"
Jul 20 09:56:55.943838 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:55.943787    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmftp" podStartSLOduration=4.252309963 podStartE2EDuration="25.943768412s" totalImagesPullingTime="21.691458449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:55.943752879 +0000 UTC m=+26.576018929" watchObservedRunningTime="2026-07-20 09:56:55.943768412 +0000 UTC m=+26.576034485"
Jul 20 09:56:56.789980 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:56.789952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:56.790362 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:56.790071    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:57.790257 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.790226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:57.790951 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.790230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:57.790951 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:57.790349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:57.790951 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:57.790440    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:57.938020 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.937986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" event={"ID":"8c74b748-ca84-4956-a79b-c56f113ae50a","Type":"ContainerStarted","Data":"d6933b96f0bd97501e2d6f4bac371980a746277e78bdd150ae1b7225dfc4c8df"}
Jul 20 09:56:57.938354 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.938324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:57.938478 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.938412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:57.939908 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.939882    2577 generic.go:356] "Generic (PLEG): container finished" podID="dec8fc27-3040-4b3f-91ea-b6696b3aae15" containerID="4c4687ba8ca052f836d420b6e3facdda1edee62867af67d2a2cc748808257ac3" exitCode=0
Jul 20 09:56:57.940006 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.939962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerDied","Data":"4c4687ba8ca052f836d420b6e3facdda1edee62867af67d2a2cc748808257ac3"}
Jul 20 09:56:57.954294 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.954135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:57.968569 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:57.968502    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb" podStartSLOduration=8.69643493 podStartE2EDuration="27.968488851s" totalImagesPullingTime="19.272053921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:56:57.966770175 +0000 UTC m=+28.599036223" watchObservedRunningTime="2026-07-20 09:56:57.968488851 +0000 UTC m=+28.600754899"
Jul 20 09:56:58.790436 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:58.790406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:58.790954 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:58.790572    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:58.942929 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:58.942903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:58.960317 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:58.960294    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:56:59.160444 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.160406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tjzjf"]
Jul 20 09:56:59.160637 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.160546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:56:59.160637 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:59.160626    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:56:59.163422 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.163396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mg256"]
Jul 20 09:56:59.163588 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.163496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:56:59.163663 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:59.163615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:56:59.164122 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.164102    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7lgzb"]
Jul 20 09:56:59.164223 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.164202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:56:59.164309 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:56:59.164292    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:56:59.946676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.946635    2577 generic.go:356] "Generic (PLEG): container finished" podID="dec8fc27-3040-4b3f-91ea-b6696b3aae15" containerID="a1931b8f420bc787758d0daed9b8f1b1a3328f0535c0adc73eaf3b1aceba2a79" exitCode=0
Jul 20 09:56:59.947209 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:56:59.946689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerDied","Data":"a1931b8f420bc787758d0daed9b8f1b1a3328f0535c0adc73eaf3b1aceba2a79"}
Jul 20 09:57:00.789888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:00.789854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:00.790044 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:00.789854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:57:00.790044 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:00.789964    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:57:00.790044 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:00.789980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:00.790165 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:00.790083    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:57:00.790165 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:00.790149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:57:01.952826 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:01.952565    2577 generic.go:356] "Generic (PLEG): container finished" podID="dec8fc27-3040-4b3f-91ea-b6696b3aae15" containerID="c1cbc4d3761403748dfff01141a1d5208f18236f272bef3a072a3f1b69244007" exitCode=0
Jul 20 09:57:01.952826 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:01.952648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerDied","Data":"c1cbc4d3761403748dfff01141a1d5208f18236f272bef3a072a3f1b69244007"}
Jul 20 09:57:02.790306 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:02.790268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:02.790306 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:02.790310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:57:02.790587 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:02.790310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:02.790587 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:02.790407    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:57:02.790587 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:02.790487    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:57:02.790746 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:02.790581    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:57:03.474570 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:03.474520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:57:03.474963 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:03.474578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:03.474963 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.474677    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:03.474963 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.474682    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:03.474963 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.474747    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret podName:acf5741f-e17e-4366-bd46-a3ec4dd047ce nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.474729867 +0000 UTC m=+66.106995906 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret") pod "global-pull-secret-syncer-tjzjf" (UID: "acf5741f-e17e-4366-bd46-a3ec4dd047ce") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:03.474963 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.474763    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.474755455 +0000 UTC m=+66.107021481 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:03.675514 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:03.675478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:03.675701 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.675672    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:03.675701 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.675693    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:03.675788 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.675706    2577 projected.go:196] Error preparing data for projected volume kube-api-access-z9jxn for pod openshift-network-diagnostics/network-check-target-7lgzb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:03.675788 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:03.675761    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn podName:62eb33ef-e544-4ab4-aa09-542e5586e957 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.675747665 +0000 UTC m=+66.308013690 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z9jxn" (UniqueName: "kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn") pod "network-check-target-7lgzb" (UID: "62eb33ef-e544-4ab4-aa09-542e5586e957") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:04.790041 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:04.789999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:04.790623 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:04.789999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:04.790623 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:04.790132    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjzjf" podUID="acf5741f-e17e-4366-bd46-a3ec4dd047ce"
Jul 20 09:57:04.790623 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:04.790005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:57:04.790623 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:04.790200    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lgzb" podUID="62eb33ef-e544-4ab4-aa09-542e5586e957"
Jul 20 09:57:04.790623 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:04.790324    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mg256" podUID="7da1e17e-2528-47b6-90a5-cc7f4d4103a8"
Jul 20 09:57:05.715818 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.715792    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-140.ec2.internal" event="NodeReady"
Jul 20 09:57:05.715968 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.715934    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 09:57:05.772612 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.772524    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8bgp6"]
Jul 20 09:57:05.789492 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.789447    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w6n4p"]
Jul 20 09:57:05.789666 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.789599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.794735 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.794713    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 09:57:05.795812 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.795512    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 09:57:05.795812 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.795638    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9txk\""
Jul 20 09:57:05.815663 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.815629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:05.820265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.819383    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8bgp6"]
Jul 20 09:57:05.820265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.819456    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w6n4p"]
Jul 20 09:57:05.820265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.819521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:05.820265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.819698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 09:57:05.820265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.819947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 09:57:05.820265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.820142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n6zqs\""
Jul 20 09:57:05.820265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.820199    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 09:57:05.823313 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.823293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27wd7\""
Jul 20 09:57:05.823440 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.823362    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:05.824288 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.824104    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:05.891095 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.891066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f39cb14-5b06-42e5-82b1-fd5918ce8120-cert\") pod \"ingress-canary-w6n4p\" (UID: \"3f39cb14-5b06-42e5-82b1-fd5918ce8120\") " pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:05.891286 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.891108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnlv\" (UniqueName: \"kubernetes.io/projected/3f39cb14-5b06-42e5-82b1-fd5918ce8120-kube-api-access-wfnlv\") pod \"ingress-canary-w6n4p\" (UID: \"3f39cb14-5b06-42e5-82b1-fd5918ce8120\") " pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:05.891551 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.891470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d996786-7f9f-493d-a44b-c7010b8f790a-config-volume\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.891684 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.891648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.891742 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.891701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d996786-7f9f-493d-a44b-c7010b8f790a-tmp-dir\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.891844 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.891740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzvr\" (UniqueName: \"kubernetes.io/projected/7d996786-7f9f-493d-a44b-c7010b8f790a-kube-api-access-jrzvr\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.992841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.992809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.992841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.992847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d996786-7f9f-493d-a44b-c7010b8f790a-tmp-dir\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.993078 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.992875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzvr\" (UniqueName: \"kubernetes.io/projected/7d996786-7f9f-493d-a44b-c7010b8f790a-kube-api-access-jrzvr\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.993078 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.992963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f39cb14-5b06-42e5-82b1-fd5918ce8120-cert\") pod \"ingress-canary-w6n4p\" (UID: \"3f39cb14-5b06-42e5-82b1-fd5918ce8120\") " pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:05.993078 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.993009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnlv\" (UniqueName: \"kubernetes.io/projected/3f39cb14-5b06-42e5-82b1-fd5918ce8120-kube-api-access-wfnlv\") pod \"ingress-canary-w6n4p\" (UID: \"3f39cb14-5b06-42e5-82b1-fd5918ce8120\") " pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:05.993078 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.993040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d996786-7f9f-493d-a44b-c7010b8f790a-config-volume\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.993251 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.993165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d996786-7f9f-493d-a44b-c7010b8f790a-tmp-dir\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.993251 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:05.993176    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:05.993312 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:05.993275    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls podName:7d996786-7f9f-493d-a44b-c7010b8f790a nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.493257557 +0000 UTC m=+37.125523589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls") pod "dns-default-8bgp6" (UID: "7d996786-7f9f-493d-a44b-c7010b8f790a") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:05.993592 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.993573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d996786-7f9f-493d-a44b-c7010b8f790a-config-volume\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:05.997401 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:05.997363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f39cb14-5b06-42e5-82b1-fd5918ce8120-cert\") pod \"ingress-canary-w6n4p\" (UID: \"3f39cb14-5b06-42e5-82b1-fd5918ce8120\") " pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:06.001086 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.001062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnlv\" (UniqueName: \"kubernetes.io/projected/3f39cb14-5b06-42e5-82b1-fd5918ce8120-kube-api-access-wfnlv\") pod \"ingress-canary-w6n4p\" (UID: \"3f39cb14-5b06-42e5-82b1-fd5918ce8120\") " pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:06.001590 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.001570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzvr\" (UniqueName: \"kubernetes.io/projected/7d996786-7f9f-493d-a44b-c7010b8f790a-kube-api-access-jrzvr\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:06.126837 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.126764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w6n4p"
Jul 20 09:57:06.444470 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.444436    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w6n4p"]
Jul 20 09:57:06.448572 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:06.448542    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f39cb14_5b06_42e5_82b1_fd5918ce8120.slice/crio-233d755315174e7d5607dda4a8e38d3d4c89a11f7c6f98a563310da742eca723 WatchSource:0}: Error finding container 233d755315174e7d5607dda4a8e38d3d4c89a11f7c6f98a563310da742eca723: Status 404 returned error can't find the container with id 233d755315174e7d5607dda4a8e38d3d4c89a11f7c6f98a563310da742eca723
Jul 20 09:57:06.496678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.496652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:06.496811 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:06.496765    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:06.496860 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:06.496817    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls podName:7d996786-7f9f-493d-a44b-c7010b8f790a nodeName:}" failed. No retries permitted until 2026-07-20 09:57:07.496803803 +0000 UTC m=+38.129069829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls") pod "dns-default-8bgp6" (UID: "7d996786-7f9f-493d-a44b-c7010b8f790a") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:06.790340 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.790303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:06.790507 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.790303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:57:06.792579 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.792556    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:06.792714 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.792604    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:06.792714 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.792604    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x8bv\""
Jul 20 09:57:06.967279 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.967245    2577 generic.go:356] "Generic (PLEG): container finished" podID="dec8fc27-3040-4b3f-91ea-b6696b3aae15" containerID="2a7884d46bcb1d3675d4b5918a5f4f738558ac5b1339815addd700b394763d41" exitCode=0
Jul 20 09:57:06.967921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.967359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerDied","Data":"2a7884d46bcb1d3675d4b5918a5f4f738558ac5b1339815addd700b394763d41"}
Jul 20 09:57:06.968747 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:06.968725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w6n4p" event={"ID":"3f39cb14-5b06-42e5-82b1-fd5918ce8120","Type":"ContainerStarted","Data":"233d755315174e7d5607dda4a8e38d3d4c89a11f7c6f98a563310da742eca723"}
Jul 20 09:57:07.505455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:07.505414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:07.505658 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:07.505595    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:07.505725 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:07.505674    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls podName:7d996786-7f9f-493d-a44b-c7010b8f790a nodeName:}" failed. No retries permitted until 2026-07-20 09:57:09.505652496 +0000 UTC m=+40.137918538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls") pod "dns-default-8bgp6" (UID: "7d996786-7f9f-493d-a44b-c7010b8f790a") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:07.978502 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:07.978462    2577 generic.go:356] "Generic (PLEG): container finished" podID="dec8fc27-3040-4b3f-91ea-b6696b3aae15" containerID="6398b66dbdfc3abcec6f768fe803b9444cf07d1e0e5a7d05a22ca51d489b8674" exitCode=0
Jul 20 09:57:07.978948 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:07.978563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerDied","Data":"6398b66dbdfc3abcec6f768fe803b9444cf07d1e0e5a7d05a22ca51d489b8674"}
Jul 20 09:57:08.982265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:08.982035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w6n4p" event={"ID":"3f39cb14-5b06-42e5-82b1-fd5918ce8120","Type":"ContainerStarted","Data":"0dea82a2784c481f6a9ed52c7c35c03ee1b6da6d2e03837132956b51e86f99fc"}
Jul 20 09:57:08.985183 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:08.985154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbc85" event={"ID":"dec8fc27-3040-4b3f-91ea-b6696b3aae15","Type":"ContainerStarted","Data":"fe188826ee686f51879a32bfd5300376fc9ee5d19dbd14f8d6f1cb5ca3bc38bd"}
Jul 20 09:57:08.998244 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:08.998203    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w6n4p" podStartSLOduration=1.933386066 podStartE2EDuration="3.998186435s" totalImagesPullingTime="2.064800369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:08.997662872 +0000 UTC m=+39.629928930" watchObservedRunningTime="2026-07-20 09:57:08.998186435 +0000 UTC m=+39.630452484"
Jul 20 09:57:09.021543 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:09.021487    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qbc85" podStartSLOduration=9.995034142 podStartE2EDuration="40.021473452s" totalImagesPullingTime="30.02643931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:09.02026783 +0000 UTC m=+39.652533885" watchObservedRunningTime="2026-07-20 09:57:09.021473452 +0000 UTC m=+39.653739499"
Jul 20 09:57:09.521758 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:09.521723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:09.521932 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:09.521886    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:09.521974 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:09.521952    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls podName:7d996786-7f9f-493d-a44b-c7010b8f790a nodeName:}" failed. No retries permitted until 2026-07-20 09:57:13.52193721 +0000 UTC m=+44.154203236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls") pod "dns-default-8bgp6" (UID: "7d996786-7f9f-493d-a44b-c7010b8f790a") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:13.551016 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:13.550975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:13.551493 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:13.551137    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:13.551493 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:13.551202    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls podName:7d996786-7f9f-493d-a44b-c7010b8f790a nodeName:}" failed. No retries permitted until 2026-07-20 09:57:21.551186496 +0000 UTC m=+52.183452522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls") pod "dns-default-8bgp6" (UID: "7d996786-7f9f-493d-a44b-c7010b8f790a") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:21.607021 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:21.606980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:21.607462 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:21.607099    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:21.607462 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:21.607153    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls podName:7d996786-7f9f-493d-a44b-c7010b8f790a nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.607138823 +0000 UTC m=+68.239404849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls") pod "dns-default-8bgp6" (UID: "7d996786-7f9f-493d-a44b-c7010b8f790a") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:29.507060 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.507024    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"]
Jul 20 09:57:29.509617 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.509600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"
Jul 20 09:57:29.511602 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.511581    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.511718 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.511643    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.511718 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.511651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zmm8x\""
Jul 20 09:57:29.520487 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.520465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"]
Jul 20 09:57:29.611623 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.611592    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"]
Jul 20 09:57:29.613456 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.613438    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f6bfd579b-gk6x5"]
Jul 20 09:57:29.613613 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.613595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.615228 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.615195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.621033 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.621012    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.621143 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.621108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 20 09:57:29.624093 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.623919    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 20 09:57:29.624093 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.623944    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.624093 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.623982    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n44pf\""
Jul 20 09:57:29.624093 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.624070    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 20 09:57:29.624337 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.624158    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.624337 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.624191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.624337 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.624302    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ht65q\""
Jul 20 09:57:29.625198 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.625182    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 20 09:57:29.625473 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.625457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 20 09:57:29.625568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.625491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 20 09:57:29.638620 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.638594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"]
Jul 20 09:57:29.659816 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.658184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f6bfd579b-gk6x5"]
Jul 20 09:57:29.667342 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.667316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qctf\" (UniqueName: \"kubernetes.io/projected/7925848d-a13e-43e9-ad35-caeb2dcc5357-kube-api-access-7qctf\") pod \"volume-data-source-validator-78cf4c6f6-dhvkq\" (UID: \"7925848d-a13e-43e9-ad35-caeb2dcc5357\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"
Jul 20 09:57:29.737303 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.737265    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-djbdl"]
Jul 20 09:57:29.739176 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.739161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.759258 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.759181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 20 09:57:29.759447 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.759270    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.759447 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.759414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hzz86\""
Jul 20 09:57:29.759447 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.759439    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 20 09:57:29.759642 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.759574    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.760252 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.760235    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b475dc65-kggx8"]
Jul 20 09:57:29.761967 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.761951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.764310 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.764290    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 20 09:57:29.767749 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qctf\" (UniqueName: \"kubernetes.io/projected/7925848d-a13e-43e9-ad35-caeb2dcc5357-kube-api-access-7qctf\") pod \"volume-data-source-validator-78cf4c6f6-dhvkq\" (UID: \"7925848d-a13e-43e9-ad35-caeb2dcc5357\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"
Jul 20 09:57:29.767839 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-stats-auth\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.767900 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdsnh\" (UniqueName: \"kubernetes.io/projected/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-kube-api-access-wdsnh\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.767900 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdwt\" (UniqueName: \"kubernetes.io/projected/0139a14f-e8e2-47c4-aa1c-8eece655bde0-kube-api-access-vjdwt\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.767900 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.768048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-default-certificate\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.768048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.768048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0139a14f-e8e2-47c4-aa1c-8eece655bde0-config\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.768048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.767987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0139a14f-e8e2-47c4-aa1c-8eece655bde0-serving-cert\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.771009 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.770994    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 20 09:57:29.771633 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.771619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 20 09:57:29.771824 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.771805    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 20 09:57:29.772402 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.772389    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x8ldn\""
Jul 20 09:57:29.776198 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.776182    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 20 09:57:29.796012 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.794714    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-djbdl"]
Jul 20 09:57:29.796012 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.794779    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b475dc65-kggx8"]
Jul 20 09:57:29.796568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.796545    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.807948 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.807918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.817845 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.817815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qctf\" (UniqueName: \"kubernetes.io/projected/7925848d-a13e-43e9-ad35-caeb2dcc5357-kube-api-access-7qctf\") pod \"volume-data-source-validator-78cf4c6f6-dhvkq\" (UID: \"7925848d-a13e-43e9-ad35-caeb2dcc5357\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"
Jul 20 09:57:29.820300 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.820285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zmm8x\""
Jul 20 09:57:29.828097 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.828078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"
Jul 20 09:57:29.868304 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-trusted-ca\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.868466 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.868466 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0139a14f-e8e2-47c4-aa1c-8eece655bde0-config\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.868466 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-config\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.868466 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-ca-trust-extracted\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0139a14f-e8e2-47c4-aa1c-8eece655bde0-serving-cert\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wvj\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-kube-api-access-t2wvj\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdsnh\" (UniqueName: \"kubernetes.io/projected/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-kube-api-access-wdsnh\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-trusted-ca\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-image-registry-private-configuration\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-serving-cert\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdwt\" (UniqueName: \"kubernetes.io/projected/0139a14f-e8e2-47c4-aa1c-8eece655bde0-kube-api-access-vjdwt\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.868798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-bound-sa-token\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.869194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-default-certificate\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.869194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-installation-pull-secrets\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.869194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzs5d\" (UniqueName: \"kubernetes.io/projected/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-kube-api-access-lzs5d\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.869194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.869194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-stats-auth\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.869194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.868996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-certificates\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.870407 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.870386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 20 09:57:29.870503 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.870404    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 20 09:57:29.871006 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.870988    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 20 09:57:29.871069 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.871026    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 20 09:57:29.871069 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.871035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 20 09:57:29.871069 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.871042    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 20 09:57:29.879394 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:29.878633    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.378603769 +0000 UTC m=+61.010869841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : configmap references non-existent config key: service-ca.crt
Jul 20 09:57:29.879553 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:29.878906    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:57:29.879553 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.878976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0139a14f-e8e2-47c4-aa1c-8eece655bde0-config\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.879553 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:29.879480    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.379445292 +0000 UTC m=+61.011711357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : secret "router-metrics-certs-default" not found
Jul 20 09:57:29.881419 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.881380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0139a14f-e8e2-47c4-aa1c-8eece655bde0-serving-cert\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.883109 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.883065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-default-certificate\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.883209 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.883108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-stats-auth\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.884571 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.883990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.887513 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.887280    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 20 09:57:29.894367 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.894337    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.895969 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.895919    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 20 09:57:29.905078 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.905050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdwt\" (UniqueName: \"kubernetes.io/projected/0139a14f-e8e2-47c4-aa1c-8eece655bde0-kube-api-access-vjdwt\") pod \"service-ca-operator-5bcbc94c78-dmmwd\" (UID: \"0139a14f-e8e2-47c4-aa1c-8eece655bde0\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.906738 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.906716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdsnh\" (UniqueName: \"kubernetes.io/projected/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-kube-api-access-wdsnh\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:29.925908 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.925880    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ht65q\""
Jul 20 09:57:29.934102 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.934078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"
Jul 20 09:57:29.959356 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.959322    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq"]
Jul 20 09:57:29.962386 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:29.962340    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7925848d_a13e_43e9_ad35_caeb2dcc5357.slice/crio-ae1f88a50341b00e1ed9ba39dc919d5741ba66e7f07544b06a89e0cb5c16a596 WatchSource:0}: Error finding container ae1f88a50341b00e1ed9ba39dc919d5741ba66e7f07544b06a89e0cb5c16a596: Status 404 returned error can't find the container with id ae1f88a50341b00e1ed9ba39dc919d5741ba66e7f07544b06a89e0cb5c16a596
Jul 20 09:57:29.969518 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.969495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-trusted-ca\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.969598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.969552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-image-registry-private-configuration\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.969598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.969584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-serving-cert\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.969677 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.969629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-bound-sa-token\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.969847 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.969828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-installation-pull-secrets\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.969926 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.969869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzs5d\" (UniqueName: \"kubernetes.io/projected/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-kube-api-access-lzs5d\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.969926 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.969914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.970026 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:29.970012    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:29.970074 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:29.970027    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b475dc65-kggx8: secret "image-registry-tls" not found
Jul 20 09:57:29.970074 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.970061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-certificates\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.970174 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:29.970081    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls podName:fe3aaec4-49c4-4877-bc18-c7a48dc8d795 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.470062022 +0000 UTC m=+61.102328062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls") pod "image-registry-6b475dc65-kggx8" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795") : secret "image-registry-tls" not found
Jul 20 09:57:29.970174 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.970143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-trusted-ca\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.970277 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.970201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-config\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.970277 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.970227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-ca-trust-extracted\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.970277 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.970258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wvj\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-kube-api-access-t2wvj\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.971095 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.970673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-trusted-ca\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.971095 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.970708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-certificates\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.971095 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.971021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-ca-trust-extracted\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.971321 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.971099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-config\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.971321 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.971225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-trusted-ca\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.972898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.972809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-serving-cert\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:29.972898 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.972854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-installation-pull-secrets\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.973634 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.973613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-image-registry-private-configuration\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.981097 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.981075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-bound-sa-token\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.981425 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.981409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wvj\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-kube-api-access-t2wvj\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:29.981578 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:29.981559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzs5d\" (UniqueName: \"kubernetes.io/projected/dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb-kube-api-access-lzs5d\") pod \"console-operator-565b7b569d-djbdl\" (UID: \"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb\") " pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:30.032631 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.032557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq" event={"ID":"7925848d-a13e-43e9-ad35-caeb2dcc5357","Type":"ContainerStarted","Data":"ae1f88a50341b00e1ed9ba39dc919d5741ba66e7f07544b06a89e0cb5c16a596"}
Jul 20 09:57:30.048925 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.048900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:30.081483 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.081449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd"]
Jul 20 09:57:30.085377 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:30.085325    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0139a14f_e8e2_47c4_aa1c_8eece655bde0.slice/crio-87ca90110edefd9da15b399ce0a87df23e0041d091c9c237ef684299460345c8 WatchSource:0}: Error finding container 87ca90110edefd9da15b399ce0a87df23e0041d091c9c237ef684299460345c8: Status 404 returned error can't find the container with id 87ca90110edefd9da15b399ce0a87df23e0041d091c9c237ef684299460345c8
Jul 20 09:57:30.176597 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.176559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-djbdl"]
Jul 20 09:57:30.179605 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:30.179577    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4c69cf_e0a0_410e_a9df_e7f0f8319ecb.slice/crio-ad328891c0eb665065c1e4716353b522d74fc68418836d7836c6d54e2fb06f3e WatchSource:0}: Error finding container ad328891c0eb665065c1e4716353b522d74fc68418836d7836c6d54e2fb06f3e: Status 404 returned error can't find the container with id ad328891c0eb665065c1e4716353b522d74fc68418836d7836c6d54e2fb06f3e
Jul 20 09:57:30.473507 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.473466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:30.473683 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.473524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:30.473683 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:30.473627    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:57:30.473683 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:30.473662    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:31.473639153 +0000 UTC m=+62.105905184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : configmap references non-existent config key: service-ca.crt
Jul 20 09:57:30.473831 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.473723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:30.473831 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:30.473800    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:31.4737829 +0000 UTC m=+62.106048925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : secret "router-metrics-certs-default" not found
Jul 20 09:57:30.473896 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:30.473845    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:30.473896 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:30.473854    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b475dc65-kggx8: secret "image-registry-tls" not found
Jul 20 09:57:30.473896 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:30.473893    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls podName:fe3aaec4-49c4-4877-bc18-c7a48dc8d795 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:31.473882226 +0000 UTC m=+62.106148272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls") pod "image-registry-6b475dc65-kggx8" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795") : secret "image-registry-tls" not found
Jul 20 09:57:30.962270 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:30.962240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h6hb"
Jul 20 09:57:31.038823 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:31.038784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" event={"ID":"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb","Type":"ContainerStarted","Data":"ad328891c0eb665065c1e4716353b522d74fc68418836d7836c6d54e2fb06f3e"}
Jul 20 09:57:31.040940 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:31.040910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd" event={"ID":"0139a14f-e8e2-47c4-aa1c-8eece655bde0","Type":"ContainerStarted","Data":"87ca90110edefd9da15b399ce0a87df23e0041d091c9c237ef684299460345c8"}
Jul 20 09:57:31.483893 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:31.483848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:31.484071 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:31.483932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:31.484071 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:31.484002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:31.484190 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:31.484166    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:33.484151632 +0000 UTC m=+64.116417674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : configmap references non-existent config key: service-ca.crt
Jul 20 09:57:31.484663 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:31.484641    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:57:31.484734 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:31.484708    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:33.484690263 +0000 UTC m=+64.116956314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : secret "router-metrics-certs-default" not found
Jul 20 09:57:31.484800 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:31.484766    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:31.484800 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:31.484775    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b475dc65-kggx8: secret "image-registry-tls" not found
Jul 20 09:57:31.484864 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:31.484805    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls podName:fe3aaec4-49c4-4877-bc18-c7a48dc8d795 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:33.48479481 +0000 UTC m=+64.117060836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls") pod "image-registry-6b475dc65-kggx8" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795") : secret "image-registry-tls" not found
Jul 20 09:57:32.045313 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:32.045271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq" event={"ID":"7925848d-a13e-43e9-ad35-caeb2dcc5357","Type":"ContainerStarted","Data":"12c3678f76abc7540b3856ee20d487c32cd8de8312631d64a0fd10be720229de"}
Jul 20 09:57:33.049475 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.049448    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/0.log"
Jul 20 09:57:33.049960 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.049487    2577 generic.go:356] "Generic (PLEG): container finished" podID="dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb" containerID="b83cf9f53ed80cc77f55253b72b7373c771fb3265261560eb2cd6c4bb05135ad" exitCode=255
Jul 20 09:57:33.049960 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.049524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" event={"ID":"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb","Type":"ContainerDied","Data":"b83cf9f53ed80cc77f55253b72b7373c771fb3265261560eb2cd6c4bb05135ad"}
Jul 20 09:57:33.049960 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.049784    2577 scope.go:117] "RemoveContainer" containerID="b83cf9f53ed80cc77f55253b72b7373c771fb3265261560eb2cd6c4bb05135ad"
Jul 20 09:57:33.051220 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.051193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd" event={"ID":"0139a14f-e8e2-47c4-aa1c-8eece655bde0","Type":"ContainerStarted","Data":"bbacb4fdd451f4bb29b00ac23f91e48bfaa44cb400b5f2587f8d3199fed9268c"}
Jul 20 09:57:33.068249 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.067809    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-dhvkq" podStartSLOduration=2.708371842 podStartE2EDuration="4.067790318s" totalImagesPullingTime="1.359418476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:32.073446384 +0000 UTC m=+62.705712434" watchObservedRunningTime="2026-07-20 09:57:33.067790318 +0000 UTC m=+63.700056369"
Jul 20 09:57:33.081655 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.081609    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd" podStartSLOduration=1.543687517 podStartE2EDuration="4.08159442s" totalImagesPullingTime="2.537906903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:33.081011968 +0000 UTC m=+63.713278019" watchObservedRunningTime="2026-07-20 09:57:33.08159442 +0000 UTC m=+63.713860467"
Jul 20 09:57:33.501508 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.501473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:33.501689 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.501555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:33.501689 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.501622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:33.501689 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:33.501673    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.5016537 +0000 UTC m=+68.133919746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : configmap references non-existent config key: service-ca.crt
Jul 20 09:57:33.501836 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:33.501716    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:33.501836 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:33.501728    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b475dc65-kggx8: secret "image-registry-tls" not found
Jul 20 09:57:33.501836 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:33.501736    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:57:33.501836 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:33.501762    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls podName:fe3aaec4-49c4-4877-bc18-c7a48dc8d795 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.501751613 +0000 UTC m=+68.134017644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls") pod "image-registry-6b475dc65-kggx8" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795") : secret "image-registry-tls" not found
Jul 20 09:57:33.501836 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:33.501801    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.501783014 +0000 UTC m=+68.134049045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : secret "router-metrics-certs-default" not found
Jul 20 09:57:33.678005 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.677969    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"]
Jul 20 09:57:33.679860 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.679839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"
Jul 20 09:57:33.682784 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.682760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 20 09:57:33.683046 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.683030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 20 09:57:33.683358 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.683345    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-95ddl\""
Jul 20 09:57:33.700568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.700538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"]
Jul 20 09:57:33.703919 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.703897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9snn\" (UniqueName: \"kubernetes.io/projected/d4dafae4-d1e1-4750-9482-793dc605e0fd-kube-api-access-d9snn\") pod \"migrator-7cc9c7ffd7-t5p2l\" (UID: \"d4dafae4-d1e1-4750-9482-793dc605e0fd\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"
Jul 20 09:57:33.805081 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.805003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9snn\" (UniqueName: \"kubernetes.io/projected/d4dafae4-d1e1-4750-9482-793dc605e0fd-kube-api-access-d9snn\") pod \"migrator-7cc9c7ffd7-t5p2l\" (UID: \"d4dafae4-d1e1-4750-9482-793dc605e0fd\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"
Jul 20 09:57:33.814071 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.814038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9snn\" (UniqueName: \"kubernetes.io/projected/d4dafae4-d1e1-4750-9482-793dc605e0fd-kube-api-access-d9snn\") pod \"migrator-7cc9c7ffd7-t5p2l\" (UID: \"d4dafae4-d1e1-4750-9482-793dc605e0fd\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"
Jul 20 09:57:33.898184 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.898152    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j7t54"]
Jul 20 09:57:33.900303 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.900283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:33.902622 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.902593    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 09:57:33.902744 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.902637    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 09:57:33.902744 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.902664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 09:57:33.902744 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.902700    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m8wfj\""
Jul 20 09:57:33.903068 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.903050    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 09:57:33.905779 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.905761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:33.905888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.905801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9cp\" (UniqueName: \"kubernetes.io/projected/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-api-access-9b9cp\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:33.905888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.905850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-crio-socket\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:33.905992 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.905887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:33.905992 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.905943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-data-volume\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:33.913748 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.913727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7t54"]
Jul 20 09:57:33.988208 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:33.988173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"
Jul 20 09:57:34.006468 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.006443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-crio-socket\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.006631 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.006496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.006631 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.006591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-crio-socket\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.006631 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.006597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-data-volume\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.006799 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.006651    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:34.006799 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.006675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.006799 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.006717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9cp\" (UniqueName: \"kubernetes.io/projected/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-api-access-9b9cp\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.006989 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.006818    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.006989 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.006882    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.506860343 +0000 UTC m=+65.139126393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.006989 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.006889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-data-volume\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.006989 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.006905    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:34.506896053 +0000 UTC m=+65.139162084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:34.019079 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.019049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9cp\" (UniqueName: \"kubernetes.io/projected/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-api-access-9b9cp\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.055688 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.055621    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 09:57:34.056163 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.056143    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/0.log"
Jul 20 09:57:34.056214 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.056192    2577 generic.go:356] "Generic (PLEG): container finished" podID="dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb" containerID="33edf8983784cda87da546c58435915c37016db2f00071c6d1f8d8af7cb77cad" exitCode=255
Jul 20 09:57:34.056289 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.056264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" event={"ID":"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb","Type":"ContainerDied","Data":"33edf8983784cda87da546c58435915c37016db2f00071c6d1f8d8af7cb77cad"}
Jul 20 09:57:34.056341 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.056324    2577 scope.go:117] "RemoveContainer" containerID="b83cf9f53ed80cc77f55253b72b7373c771fb3265261560eb2cd6c4bb05135ad"
Jul 20 09:57:34.056628 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.056592    2577 scope.go:117] "RemoveContainer" containerID="33edf8983784cda87da546c58435915c37016db2f00071c6d1f8d8af7cb77cad"
Jul 20 09:57:34.056898 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.056815    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-djbdl_openshift-console-operator(dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb)\"" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" podUID="dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb"
Jul 20 09:57:34.137841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.137814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l"]
Jul 20 09:57:34.141094 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:34.141065    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dafae4_d1e1_4750_9482_793dc605e0fd.slice/crio-b6e97107cc8c776c8db2621563a76a670bd6b2d2cec2cdce50ac0cf0062fe031 WatchSource:0}: Error finding container b6e97107cc8c776c8db2621563a76a670bd6b2d2cec2cdce50ac0cf0062fe031: Status 404 returned error can't find the container with id b6e97107cc8c776c8db2621563a76a670bd6b2d2cec2cdce50ac0cf0062fe031
Jul 20 09:57:34.510891 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.510857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.511051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:34.510962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:34.511051 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.511003    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.511119 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.511082    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.511065822 +0000 UTC m=+66.143331850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:34.511119 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.511085    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:34.511195 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:34.511139    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.511127434 +0000 UTC m=+66.143393460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:35.060841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.060808    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 09:57:35.061223 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.061159    2577 scope.go:117] "RemoveContainer" containerID="33edf8983784cda87da546c58435915c37016db2f00071c6d1f8d8af7cb77cad"
Jul 20 09:57:35.061350 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:35.061326    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-djbdl_openshift-console-operator(dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb)\"" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" podUID="dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb"
Jul 20 09:57:35.062065 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.062041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l" event={"ID":"d4dafae4-d1e1-4750-9482-793dc605e0fd","Type":"ContainerStarted","Data":"b6e97107cc8c776c8db2621563a76a670bd6b2d2cec2cdce50ac0cf0062fe031"}
Jul 20 09:57:35.520008 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.519975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:35.520008 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.520018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:57:35.520239 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:35.520105    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:35.520239 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:35.520172    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.520158481 +0000 UTC m=+68.152424507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:35.520239 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.520228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:35.520352 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.520252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:35.520400 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:35.520381    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:35.520459 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:35.520447    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:37.520431628 +0000 UTC m=+68.152697688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:35.531956 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.531935    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:35.536699 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.536685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:35.540308 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:35.540294    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 09:57:35.540359 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:35.540348    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs podName:7da1e17e-2528-47b6-90a5-cc7f4d4103a8 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:39.540331685 +0000 UTC m=+130.172597711 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs") pod "network-metrics-daemon-mg256" (UID: "7da1e17e-2528-47b6-90a5-cc7f4d4103a8") : secret "metrics-daemon-secret" not found
Jul 20 09:57:35.542682 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.542666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acf5741f-e17e-4366-bd46-a3ec4dd047ce-original-pull-secret\") pod \"global-pull-secret-syncer-tjzjf\" (UID: \"acf5741f-e17e-4366-bd46-a3ec4dd047ce\") " pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:35.600074 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.599991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjzjf"
Jul 20 09:57:35.721663 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.721635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:35.722937 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.722910    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tjzjf"]
Jul 20 09:57:35.723839 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.723820    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:35.726316 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:35.726291    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf5741f_e17e_4366_bd46_a3ec4dd047ce.slice/crio-ec79c81ceffcda941d8578eff3a91c43e4788e18221bb21f602a4e42d8d2175a WatchSource:0}: Error finding container ec79c81ceffcda941d8578eff3a91c43e4788e18221bb21f602a4e42d8d2175a: Status 404 returned error can't find the container with id ec79c81ceffcda941d8578eff3a91c43e4788e18221bb21f602a4e42d8d2175a
Jul 20 09:57:35.734692 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.734666    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:35.746288 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.746264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jxn\" (UniqueName: \"kubernetes.io/projected/62eb33ef-e544-4ab4-aa09-542e5586e957-kube-api-access-z9jxn\") pod \"network-check-target-7lgzb\" (UID: \"62eb33ef-e544-4ab4-aa09-542e5586e957\") " pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:35.836028 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.836002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-27wd7\""
Jul 20 09:57:35.844064 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.844039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:35.969643 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:35.969607    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7lgzb"]
Jul 20 09:57:35.973377 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:35.973354    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62eb33ef_e544_4ab4_aa09_542e5586e957.slice/crio-04825c1d92123870d81bdd54b711997ab1ddbc44e57b70d47f69c2e28b1085c9 WatchSource:0}: Error finding container 04825c1d92123870d81bdd54b711997ab1ddbc44e57b70d47f69c2e28b1085c9: Status 404 returned error can't find the container with id 04825c1d92123870d81bdd54b711997ab1ddbc44e57b70d47f69c2e28b1085c9
Jul 20 09:57:36.068041 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:36.068003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7lgzb" event={"ID":"62eb33ef-e544-4ab4-aa09-542e5586e957","Type":"ContainerStarted","Data":"04825c1d92123870d81bdd54b711997ab1ddbc44e57b70d47f69c2e28b1085c9"}
Jul 20 09:57:36.069209 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:36.069185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tjzjf" event={"ID":"acf5741f-e17e-4366-bd46-a3ec4dd047ce","Type":"ContainerStarted","Data":"ec79c81ceffcda941d8578eff3a91c43e4788e18221bb21f602a4e42d8d2175a"}
Jul 20 09:57:36.070758 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:36.070736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l" event={"ID":"d4dafae4-d1e1-4750-9482-793dc605e0fd","Type":"ContainerStarted","Data":"53ec815c36f2b36ec611280fbbb29547363f4f05da758e12bc39348c8595a614"}
Jul 20 09:57:36.070865 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:36.070770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l" event={"ID":"d4dafae4-d1e1-4750-9482-793dc605e0fd","Type":"ContainerStarted","Data":"3aa9173ecee018748aef1d74e269d6e4444270385fad1c1df2ce331aafd5c509"}
Jul 20 09:57:36.086856 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:36.086807    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-t5p2l" podStartSLOduration=1.943070803 podStartE2EDuration="3.086791925s" totalImagesPullingTime="1.143721122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:36.085573394 +0000 UTC m=+66.717839444" watchObservedRunningTime="2026-07-20 09:57:36.086791925 +0000 UTC m=+66.719058014"
Jul 20 09:57:36.287680 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:36.287654    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wmz4_29c3298f-7d97-47d3-a41a-b064e8935ff1/dns-node-resolver/0.log"
Jul 20 09:57:37.488971 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:37.488938    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x48x8_e0a99480-abae-43bb-86a7-47ef8e6836d2/node-ca/0.log"
Jul 20 09:57:37.537348 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:37.537315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:37.537510 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:37.537387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:37.537510 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:37.537414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:37.537510 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:37.537454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:37.537510 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:37.537478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:37.537510 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537490    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 09:57:37.537758 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537557    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b475dc65-kggx8: secret "image-registry-tls" not found
Jul 20 09:57:37.537758 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537642    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:37.537758 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537657    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls podName:fe3aaec4-49c4-4877-bc18-c7a48dc8d795 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:45.537631154 +0000 UTC m=+76.169897183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls") pod "image-registry-6b475dc65-kggx8" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795") : secret "image-registry-tls" not found
Jul 20 09:57:37.537758 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537692    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:41.53767745 +0000 UTC m=+72.169943476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:37.537758 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537745    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:45.537738872 +0000 UTC m=+76.170004897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : configmap references non-existent config key: service-ca.crt
Jul 20 09:57:37.537928 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537783    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:57:37.537928 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537801    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:45.537795672 +0000 UTC m=+76.170061697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : secret "router-metrics-certs-default" not found
Jul 20 09:57:37.537928 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537571    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:37.537928 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.537822    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:41.537817229 +0000 UTC m=+72.170083254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:37.638227 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:37.638194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:57:37.638380 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.638367    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:37.638450 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:37.638435    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls podName:7d996786-7f9f-493d-a44b-c7010b8f790a nodeName:}" failed. No retries permitted until 2026-07-20 09:58:09.63841629 +0000 UTC m=+100.270682331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls") pod "dns-default-8bgp6" (UID: "7d996786-7f9f-493d-a44b-c7010b8f790a") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:38.089906 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:38.089870    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w6n4p_3f39cb14-5b06-42e5-82b1-fd5918ce8120/serve-healthcheck-canary/0.log"
Jul 20 09:57:38.288343 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:38.288300    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-t5p2l_d4dafae4-d1e1-4750-9482-793dc605e0fd/migrator/0.log"
Jul 20 09:57:38.492145 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:38.492112    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-t5p2l_d4dafae4-d1e1-4750-9482-793dc605e0fd/graceful-termination/0.log"
Jul 20 09:57:40.049750 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:40.049709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:40.050247 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:40.049791    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:40.050247 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:40.050233    2577 scope.go:117] "RemoveContainer" containerID="33edf8983784cda87da546c58435915c37016db2f00071c6d1f8d8af7cb77cad"
Jul 20 09:57:40.050472 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:40.050448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-djbdl_openshift-console-operator(dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb)\"" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" podUID="dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb"
Jul 20 09:57:41.090823 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:41.090791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7lgzb" event={"ID":"62eb33ef-e544-4ab4-aa09-542e5586e957","Type":"ContainerStarted","Data":"41c9d140f6076439b6ab2169b1c6ecafba197bc7ebaa17a00e4fa4534df38ec8"}
Jul 20 09:57:41.091291 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:41.090918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:57:41.092410 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:41.092379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tjzjf" event={"ID":"acf5741f-e17e-4366-bd46-a3ec4dd047ce","Type":"ContainerStarted","Data":"4616626d2c929c59a153ec1c7bf2ba1f184208776ad139a4a4a11a012dbfa00d"}
Jul 20 09:57:41.111118 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:41.111074    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7lgzb" podStartSLOduration=66.411594553 podStartE2EDuration="1m11.111061586s" totalImagesPullingTime="4.699467033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:41.110210273 +0000 UTC m=+71.742476324" watchObservedRunningTime="2026-07-20 09:57:41.111061586 +0000 UTC m=+71.743327659"
Jul 20 09:57:41.129716 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:41.129671    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tjzjf" podStartSLOduration=66.169950717 podStartE2EDuration="1m11.129656924s" totalImagesPullingTime="4.959706207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:41.128822098 +0000 UTC m=+71.761088147" watchObservedRunningTime="2026-07-20 09:57:41.129656924 +0000 UTC m=+71.761922972"
Jul 20 09:57:41.574244 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:41.574208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:41.574412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:41.574291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:41.574412 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:41.574343    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:41.574412 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:41.574357    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:57:41.574412 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:41.574405    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:49.574389696 +0000 UTC m=+80.206655722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:41.574614 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:41.574418    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:49.574412214 +0000 UTC m=+80.206678240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : configmap "kube-rbac-proxy" not found
Jul 20 09:57:45.605389 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:45.605347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:45.605915 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:45.605417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:57:45.605915 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:45.605463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:45.605915 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:45.605508    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:01.605489057 +0000 UTC m=+92.237755088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : configmap references non-existent config key: service-ca.crt
Jul 20 09:57:45.605915 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:45.605597    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 20 09:57:45.605915 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:45.605651    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs podName:cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:01.605633519 +0000 UTC m=+92.237899570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs") pod "router-default-5f6bfd579b-gk6x5" (UID: "cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1") : secret "router-metrics-certs-default" not found
Jul 20 09:57:45.608008 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:45.607990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"image-registry-6b475dc65-kggx8\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") " pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:45.670594 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:45.670558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:45.807752 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:45.807719    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b475dc65-kggx8"]
Jul 20 09:57:45.811228 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:57:45.811191    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3aaec4_49c4_4877_bc18_c7a48dc8d795.slice/crio-c7fc7bb70266f51099555171c51f23290820caf0eb015323b1c54f0b8130e036 WatchSource:0}: Error finding container c7fc7bb70266f51099555171c51f23290820caf0eb015323b1c54f0b8130e036: Status 404 returned error can't find the container with id c7fc7bb70266f51099555171c51f23290820caf0eb015323b1c54f0b8130e036
Jul 20 09:57:46.108263 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:46.108230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" event={"ID":"fe3aaec4-49c4-4877-bc18-c7a48dc8d795","Type":"ContainerStarted","Data":"82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d"}
Jul 20 09:57:46.108453 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:46.108279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" event={"ID":"fe3aaec4-49c4-4877-bc18-c7a48dc8d795","Type":"ContainerStarted","Data":"c7fc7bb70266f51099555171c51f23290820caf0eb015323b1c54f0b8130e036"}
Jul 20 09:57:46.108453 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:46.108331    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:57:46.134980 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:46.134873    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" podStartSLOduration=17.134844607 podStartE2EDuration="17.134844607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:46.133585931 +0000 UTC m=+76.765851980" watchObservedRunningTime="2026-07-20 09:57:46.134844607 +0000 UTC m=+76.767110655"
Jul 20 09:57:49.638027 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:49.637988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:49.638379 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:49.638059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:49.638379 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:49.638135    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:49.638379 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:57:49.638203    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls podName:55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:05.638187766 +0000 UTC m=+96.270453804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7t54" (UID: "55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:57:49.638550 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:49.638513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:57:53.790489 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:53.790375    2577 scope.go:117] "RemoveContainer" containerID="33edf8983784cda87da546c58435915c37016db2f00071c6d1f8d8af7cb77cad"
Jul 20 09:57:54.134475 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:54.134400    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 09:57:54.134619 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:54.134479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" event={"ID":"dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb","Type":"ContainerStarted","Data":"c651ca834f30ba7393beae06f03de731a3e88503ac58675552e5e48cc8e1c43a"}
Jul 20 09:57:54.134788 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:54.134766    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:57:54.175940 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:54.175876    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-djbdl" podStartSLOduration=22.733484818 podStartE2EDuration="25.175856667s" totalImagesPullingTime="2.442371849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:54.174945479 +0000 UTC m=+84.807211528" watchObservedRunningTime="2026-07-20 09:57:54.175856667 +0000 UTC m=+84.808122720"
Jul 20 09:57:54.709094 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:57:54.709061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-djbdl"
Jul 20 09:58:00.134135 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:00.134106    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b475dc65-kggx8"]
Jul 20 09:58:00.139549 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:00.139498    2577 patch_prober.go:28] interesting pod/image-registry-6b475dc65-kggx8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 20 09:58:00.139674 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:00.139577    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" podUID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 09:58:01.637201 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:01.637163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:01.637672 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:01.637223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:01.637840 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:01.637820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-service-ca-bundle\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:01.639746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:01.639725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1-metrics-certs\") pod \"router-default-5f6bfd579b-gk6x5\" (UID: \"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1\") " pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:01.730844 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:01.730812    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n44pf\""
Jul 20 09:58:01.739610 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:01.739584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:01.874253 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:01.874211    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f6bfd579b-gk6x5"]
Jul 20 09:58:01.877299 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:01.877263    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfddf79e_8aa4_4d31_98e5_6c107ff6a1e1.slice/crio-9aa0a122fb1d4c546580d11b256118cc642cf9af6c74048e1d9d2995675df2f6 WatchSource:0}: Error finding container 9aa0a122fb1d4c546580d11b256118cc642cf9af6c74048e1d9d2995675df2f6: Status 404 returned error can't find the container with id 9aa0a122fb1d4c546580d11b256118cc642cf9af6c74048e1d9d2995675df2f6
Jul 20 09:58:02.163251 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:02.163156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5" event={"ID":"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1","Type":"ContainerStarted","Data":"d5c7935363c29c1a2feb5245838192e3539cccf9b3a554d08fee4452b5110763"}
Jul 20 09:58:02.163251 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:02.163205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5" event={"ID":"cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1","Type":"ContainerStarted","Data":"9aa0a122fb1d4c546580d11b256118cc642cf9af6c74048e1d9d2995675df2f6"}
Jul 20 09:58:02.182071 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:02.182017    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5" podStartSLOduration=33.181997831 podStartE2EDuration="33.181997831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:02.180832761 +0000 UTC m=+92.813098809" watchObservedRunningTime="2026-07-20 09:58:02.181997831 +0000 UTC m=+92.814263879"
Jul 20 09:58:02.740606 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:02.740567    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:02.743060 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:02.743037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:03.166435 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:03.166361    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:03.167664 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:03.167640    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f6bfd579b-gk6x5"
Jul 20 09:58:05.666491 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:05.666454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:58:05.673322 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:05.673299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7t54\" (UID: \"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70\") " pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:58:05.710249 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:05.710223    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7t54"
Jul 20 09:58:05.848983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:05.848950    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7t54"]
Jul 20 09:58:05.851993 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:05.851965    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a9ae5e_7b1a_47de_b3e9_b036e1cd0e70.slice/crio-c0cbc0f040c8a6cb1340e4182456016de4f803a19a2bcf118510c115c3dc44bb WatchSource:0}: Error finding container c0cbc0f040c8a6cb1340e4182456016de4f803a19a2bcf118510c115c3dc44bb: Status 404 returned error can't find the container with id c0cbc0f040c8a6cb1340e4182456016de4f803a19a2bcf118510c115c3dc44bb
Jul 20 09:58:06.179090 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:06.179060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7t54" event={"ID":"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70","Type":"ContainerStarted","Data":"c0cbc0f040c8a6cb1340e4182456016de4f803a19a2bcf118510c115c3dc44bb"}
Jul 20 09:58:09.705238 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:09.705195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:58:09.707899 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:09.707872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d996786-7f9f-493d-a44b-c7010b8f790a-metrics-tls\") pod \"dns-default-8bgp6\" (UID: \"7d996786-7f9f-493d-a44b-c7010b8f790a\") " pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:58:10.004024 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.003941    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9txk\""
Jul 20 09:58:10.011946 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.011925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:58:10.139331 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.139296    2577 patch_prober.go:28] interesting pod/image-registry-6b475dc65-kggx8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 20 09:58:10.139477 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.139354    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" podUID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 09:58:10.806973 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.806934    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59497659dd-46q5x"]
Jul 20 09:58:10.826915 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.826873    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59497659dd-46q5x"]
Jul 20 09:58:10.827067 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.827021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:10.831052 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.830705    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 20 09:58:10.831052 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.830744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 09:58:10.831052 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.830768    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 09:58:10.831052 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.830781    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 20 09:58:10.831052 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.830809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 09:58:10.831052 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.830705    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 09:58:10.831430 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.831156    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nmtb\""
Jul 20 09:58:10.831430 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.831245    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 09:58:10.915429 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.915400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-serving-cert\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:10.915429 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.915437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-oauth-config\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:10.915681 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.915459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-config\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:10.915681 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.915520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-oauth-serving-cert\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:10.915681 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.915637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrp4\" (UniqueName: \"kubernetes.io/projected/14b5e2e7-b679-4833-bb68-7c7ee01a135e-kube-api-access-ndrp4\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:10.915827 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:10.915725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-service-ca\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.016696 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.016652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-serving-cert\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.016848 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.016707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-oauth-config\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.016848 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.016745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-config\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.016848 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.016774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-oauth-serving-cert\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.016848 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.016813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrp4\" (UniqueName: \"kubernetes.io/projected/14b5e2e7-b679-4833-bb68-7c7ee01a135e-kube-api-access-ndrp4\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.017068 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.016870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-service-ca\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.017574 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.017522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-oauth-serving-cert\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.017706 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.017683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-service-ca\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.017765 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.017723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-config\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.019298 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.019273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-oauth-config\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.019516 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.019497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-serving-cert\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.025915 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.025893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrp4\" (UniqueName: \"kubernetes.io/projected/14b5e2e7-b679-4833-bb68-7c7ee01a135e-kube-api-access-ndrp4\") pod \"console-59497659dd-46q5x\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") " pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.036843 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.036824    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8bgp6"]
Jul 20 09:58:11.040355 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:11.040331    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d996786_7f9f_493d_a44b_c7010b8f790a.slice/crio-3d0bfc6532be19e5a1e93b6b3620339455f333b620c56f4f83898ab28e051d82 WatchSource:0}: Error finding container 3d0bfc6532be19e5a1e93b6b3620339455f333b620c56f4f83898ab28e051d82: Status 404 returned error can't find the container with id 3d0bfc6532be19e5a1e93b6b3620339455f333b620c56f4f83898ab28e051d82
Jul 20 09:58:11.139810 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.139736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:11.196872 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.196839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8bgp6" event={"ID":"7d996786-7f9f-493d-a44b-c7010b8f790a","Type":"ContainerStarted","Data":"3d0bfc6532be19e5a1e93b6b3620339455f333b620c56f4f83898ab28e051d82"}
Jul 20 09:58:11.392147 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:11.392063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59497659dd-46q5x"]
Jul 20 09:58:11.395400 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:11.395366    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b5e2e7_b679_4833_bb68_7c7ee01a135e.slice/crio-01d9df678f285c6ca072dee3c71e27a938030702ea7501ab458fba6da5824026 WatchSource:0}: Error finding container 01d9df678f285c6ca072dee3c71e27a938030702ea7501ab458fba6da5824026: Status 404 returned error can't find the container with id 01d9df678f285c6ca072dee3c71e27a938030702ea7501ab458fba6da5824026
Jul 20 09:58:12.113804 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:12.113188    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7lgzb"
Jul 20 09:58:12.203352 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:12.203312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7t54" event={"ID":"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70","Type":"ContainerStarted","Data":"baa63dfcabfd73f272d7ebaef093173a352e7067f8dea230b4d5d8c34d5bcbf5"}
Jul 20 09:58:12.205212 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:12.205141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59497659dd-46q5x" event={"ID":"14b5e2e7-b679-4833-bb68-7c7ee01a135e","Type":"ContainerStarted","Data":"01d9df678f285c6ca072dee3c71e27a938030702ea7501ab458fba6da5824026"}
Jul 20 09:58:14.215127 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:14.215087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8bgp6" event={"ID":"7d996786-7f9f-493d-a44b-c7010b8f790a","Type":"ContainerStarted","Data":"9c368cfd81896d4de8a7807cd049c7525532e63d0d3e560d74b4b45ff9110a77"}
Jul 20 09:58:17.228013 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:17.227974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7t54" event={"ID":"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70","Type":"ContainerStarted","Data":"ac38e4f3a57ed51d914b2e669077133701282a19606055925c7258c0fd1f3b4d"}
Jul 20 09:58:17.230391 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:17.229888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8bgp6" event={"ID":"7d996786-7f9f-493d-a44b-c7010b8f790a","Type":"ContainerStarted","Data":"5741d442faaee5663993736105c270fe4e6eadd76ff9530382f67bc4b1a5ad42"}
Jul 20 09:58:17.230391 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:17.230006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:58:17.231546 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:17.231498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59497659dd-46q5x" event={"ID":"14b5e2e7-b679-4833-bb68-7c7ee01a135e","Type":"ContainerStarted","Data":"99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8"}
Jul 20 09:58:17.248497 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:17.248418    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8bgp6" podStartSLOduration=69.974851836 podStartE2EDuration="1m12.248399519s" totalImagesPullingTime="2.273547683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:17.246994824 +0000 UTC m=+107.879260874" watchObservedRunningTime="2026-07-20 09:58:17.248399519 +0000 UTC m=+107.880665568"
Jul 20 09:58:17.264386 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:17.264328    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59497659dd-46q5x" podStartSLOduration=2.256414007 podStartE2EDuration="7.264307232s" totalImagesPullingTime="5.007893225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:17.263401489 +0000 UTC m=+107.895667538" watchObservedRunningTime="2026-07-20 09:58:17.264307232 +0000 UTC m=+107.896573281"
Jul 20 09:58:19.399112 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.399081    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7978db7485-jmw9s"]
Jul 20 09:58:19.409060 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.409033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.413170 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.413140    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7978db7485-jmw9s"]
Jul 20 09:58:19.417755 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.417726    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 09:58:19.490670 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.490638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-serving-cert\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.490836 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.490710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-trusted-ca-bundle\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.490836 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.490752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-oauth-config\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.490836 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.490771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh2hg\" (UniqueName: \"kubernetes.io/projected/32d0e7fa-ab8e-4cab-8378-0536e8b40032-kube-api-access-fh2hg\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.490836 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.490800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-oauth-serving-cert\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.491071 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.490878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-config\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.491071 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.490943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-service-ca\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.591519 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.591486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-oauth-config\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.591713 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.591524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh2hg\" (UniqueName: \"kubernetes.io/projected/32d0e7fa-ab8e-4cab-8378-0536e8b40032-kube-api-access-fh2hg\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.591786 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.591718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-oauth-serving-cert\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.591786 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.591756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-config\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.591916 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.591794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-service-ca\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.591916 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.591826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-serving-cert\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.591916 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.591895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-trusted-ca-bundle\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.592607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.592485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-config\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.592607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.592485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-oauth-serving-cert\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.592607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.592571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-service-ca\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.592885 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.592849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-trusted-ca-bundle\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.594485 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.594450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-oauth-config\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.594961 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.594936    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-serving-cert\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.599780 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.599759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh2hg\" (UniqueName: \"kubernetes.io/projected/32d0e7fa-ab8e-4cab-8378-0536e8b40032-kube-api-access-fh2hg\") pod \"console-7978db7485-jmw9s\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") " pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.720611 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.720571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:19.960501 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:19.960473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7978db7485-jmw9s"]
Jul 20 09:58:19.963493 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:19.963470    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d0e7fa_ab8e_4cab_8378_0536e8b40032.slice/crio-9f1cd72c092b8ff7250ca0dcc1f558b4328c410ac9fd032688d01dadf76f02c4 WatchSource:0}: Error finding container 9f1cd72c092b8ff7250ca0dcc1f558b4328c410ac9fd032688d01dadf76f02c4: Status 404 returned error can't find the container with id 9f1cd72c092b8ff7250ca0dcc1f558b4328c410ac9fd032688d01dadf76f02c4
Jul 20 09:58:20.140145 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:20.140116    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:58:20.245339 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:20.245239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7t54" event={"ID":"55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70","Type":"ContainerStarted","Data":"017cc73ecdd077d43aafdee9e212ea1fd9bab1bf1f75f80cb581cbcfebfbb944"}
Jul 20 09:58:20.246697 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:20.246672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7978db7485-jmw9s" event={"ID":"32d0e7fa-ab8e-4cab-8378-0536e8b40032","Type":"ContainerStarted","Data":"363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4"}
Jul 20 09:58:20.246785 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:20.246715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7978db7485-jmw9s" event={"ID":"32d0e7fa-ab8e-4cab-8378-0536e8b40032","Type":"ContainerStarted","Data":"9f1cd72c092b8ff7250ca0dcc1f558b4328c410ac9fd032688d01dadf76f02c4"}
Jul 20 09:58:20.267029 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:20.264267    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j7t54" podStartSLOduration=33.420864464 podStartE2EDuration="47.264247337s" totalImagesPullingTime="13.843382873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:57:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:20.263190005 +0000 UTC m=+110.895456053" watchObservedRunningTime="2026-07-20 09:58:20.264247337 +0000 UTC m=+110.896513386"
Jul 20 09:58:20.281264 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:20.281211    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7978db7485-jmw9s" podStartSLOduration=1.281196738 podStartE2EDuration="1.281196738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:20.279784621 +0000 UTC m=+110.912050670" watchObservedRunningTime="2026-07-20 09:58:20.281196738 +0000 UTC m=+110.913462787"
Jul 20 09:58:21.140624 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:21.140589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:21.141101 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:21.140673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:21.145322 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:21.145302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:21.254140 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:21.254106    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:24.237283 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:24.237255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8bgp6"
Jul 20 09:58:25.155903 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.155867    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" podUID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" containerName="registry" containerID="cri-o://82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d" gracePeriod=30
Jul 20 09:58:25.397385 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.397362    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:58:25.435523 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-bound-sa-token\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435523 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-installation-pull-secrets\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435523 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435511    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-certificates\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435761 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435550    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-image-registry-private-configuration\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435761 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435737    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435864 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-ca-trust-extracted\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435864 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-trusted-ca\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435865    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2wvj\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-kube-api-access-t2wvj\") pod \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\" (UID: \"fe3aaec4-49c4-4877-bc18-c7a48dc8d795\") "
Jul 20 09:58:25.435983 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.435908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:58:25.436126 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.436106    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-certificates\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:25.436588 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.436522    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:58:25.438264 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.438235    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:58:25.438370 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.438308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:58:25.438446 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.438403    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:58:25.438504 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.438445    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:58:25.438703 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.438680    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-kube-api-access-t2wvj" (OuterVolumeSpecName: "kube-api-access-t2wvj") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "kube-api-access-t2wvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:58:25.444806 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.444779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fe3aaec4-49c4-4877-bc18-c7a48dc8d795" (UID: "fe3aaec4-49c4-4877-bc18-c7a48dc8d795"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:58:25.537154 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.537118    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-bound-sa-token\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:25.537154 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.537146    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-installation-pull-secrets\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:25.537154 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.537156    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-image-registry-private-configuration\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:25.537359 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.537166    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-registry-tls\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:25.537359 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.537174    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-ca-trust-extracted\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:25.537359 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.537183    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-trusted-ca\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:25.537359 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:25.537191    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2wvj\" (UniqueName: \"kubernetes.io/projected/fe3aaec4-49c4-4877-bc18-c7a48dc8d795-kube-api-access-t2wvj\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:26.267574 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.267521    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" containerID="82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d" exitCode=0
Jul 20 09:58:26.267730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.267601    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b475dc65-kggx8"
Jul 20 09:58:26.267730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.267614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" event={"ID":"fe3aaec4-49c4-4877-bc18-c7a48dc8d795","Type":"ContainerDied","Data":"82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d"}
Jul 20 09:58:26.267730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.267654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b475dc65-kggx8" event={"ID":"fe3aaec4-49c4-4877-bc18-c7a48dc8d795","Type":"ContainerDied","Data":"c7fc7bb70266f51099555171c51f23290820caf0eb015323b1c54f0b8130e036"}
Jul 20 09:58:26.267730 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.267674    2577 scope.go:117] "RemoveContainer" containerID="82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d"
Jul 20 09:58:26.277855 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.277833    2577 scope.go:117] "RemoveContainer" containerID="82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d"
Jul 20 09:58:26.278152 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:26.278129    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d\": container with ID starting with 82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d not found: ID does not exist" containerID="82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d"
Jul 20 09:58:26.278208 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.278162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d"} err="failed to get container status \"82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d\": rpc error: code = NotFound desc = could not find container \"82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d\": container with ID starting with 82e511e00d05108df983048fe01250b25b8786f5405fa193eb2d7244bee7077d not found: ID does not exist"
Jul 20 09:58:26.281729 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.281708    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b475dc65-kggx8"]
Jul 20 09:58:26.285268 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:26.285247    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b475dc65-kggx8"]
Jul 20 09:58:27.332479 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.332448    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"]
Jul 20 09:58:27.332872 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.332696    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" containerName="registry"
Jul 20 09:58:27.332872 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.332707    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" containerName="registry"
Jul 20 09:58:27.332872 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.332759    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" containerName="registry"
Jul 20 09:58:27.337475 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.337455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.340944 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.340921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 20 09:58:27.340944 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.340927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 20 09:58:27.340944 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.340949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 09:58:27.341517 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.341502    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g6wjh\""
Jul 20 09:58:27.341609 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.341518    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 09:58:27.341609 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.341558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 09:58:27.346376 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.346356    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"]
Jul 20 09:58:27.349485 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.349467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.350188 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.350168    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"]
Jul 20 09:58:27.350422 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.350385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.350472 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.350441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.350512 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.350473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcth\" (UniqueName: \"kubernetes.io/projected/60023925-65ad-470d-aa92-bfe998e9e857-kube-api-access-cjcth\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.350615 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.350596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60023925-65ad-470d-aa92-bfe998e9e857-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.353104 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.353079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 20 09:58:27.353186 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.353120    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 20 09:58:27.353245 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.353189    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-glbc2\""
Jul 20 09:58:27.353245 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.353085    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 20 09:58:27.362461 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.362442    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nqbpx"]
Jul 20 09:58:27.366171 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.365933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.366684 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.366662    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"]
Jul 20 09:58:27.368203 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.368181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2kv7c\""
Jul 20 09:58:27.368203 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.368194    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 09:58:27.368454 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.368429    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 09:58:27.368614 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.368598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 09:58:27.451975 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.451946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731c943c-9af0-4cdd-a209-1c8d40defcf1-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.451975 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.451980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-wtmp\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452190 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.452190 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.452190 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzq6x\" (UniqueName: \"kubernetes.io/projected/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-api-access-fzq6x\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.452190 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-textfile\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452190 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452345 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-sys\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452345 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-tls\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452345 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.452345 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.452345 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-root\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452345 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b315ab5e-5d34-4c17-be7f-777549b6ce3a-metrics-client-ca\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcth\" (UniqueName: \"kubernetes.io/projected/60023925-65ad-470d-aa92-bfe998e9e857-kube-api-access-cjcth\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/731c943c-9af0-4cdd-a209-1c8d40defcf1-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:27.452435    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhnc\" (UniqueName: \"kubernetes.io/projected/b315ab5e-5d34-4c17-be7f-777549b6ce3a-kube-api-access-mbhnc\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:27.452498    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-tls podName:60023925-65ad-470d-aa92-bfe998e9e857 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:27.952482438 +0000 UTC m=+118.584748465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-24hlw" (UID: "60023925-65ad-470d-aa92-bfe998e9e857") : secret "openshift-state-metrics-tls" not found
Jul 20 09:58:27.452638 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.452512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60023925-65ad-470d-aa92-bfe998e9e857-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.453071 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.453052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60023925-65ad-470d-aa92-bfe998e9e857-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.454783 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.454753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.462448 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.462430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcth\" (UniqueName: \"kubernetes.io/projected/60023925-65ad-470d-aa92-bfe998e9e857-kube-api-access-cjcth\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.553356 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzq6x\" (UniqueName: \"kubernetes.io/projected/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-api-access-fzq6x\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.553550 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-textfile\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553550 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553550 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-sys\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553550 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-sys\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553767 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-tls\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553767 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.553767 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-root\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553767 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b315ab5e-5d34-4c17-be7f-777549b6ce3a-metrics-client-ca\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553767 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.553767 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/731c943c-9af0-4cdd-a209-1c8d40defcf1-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-root\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-textfile\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:27.553791    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhnc\" (UniqueName: \"kubernetes.io/projected/b315ab5e-5d34-4c17-be7f-777549b6ce3a-kube-api-access-mbhnc\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731c943c-9af0-4cdd-a209-1c8d40defcf1-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:27.553796    2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:27.553877    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-tls podName:b315ab5e-5d34-4c17-be7f-777549b6ce3a nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.05385641 +0000 UTC m=+118.686122459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-tls") pod "node-exporter-nqbpx" (UID: "b315ab5e-5d34-4c17-be7f-777549b6ce3a") : secret "node-exporter-tls" not found
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-wtmp\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:27.553928    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-tls podName:731c943c-9af0-4cdd-a209-1c8d40defcf1 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.053913023 +0000 UTC m=+118.686179048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-jlgj7" (UID: "731c943c-9af0-4cdd-a209-1c8d40defcf1") : secret "kube-state-metrics-tls" not found
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.553994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.554048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.554025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-wtmp\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.554682 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.554249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.554682 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.554265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b315ab5e-5d34-4c17-be7f-777549b6ce3a-metrics-client-ca\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.554682 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.554590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/731c943c-9af0-4cdd-a209-1c8d40defcf1-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.554682 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.554615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.554849 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.554727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/731c943c-9af0-4cdd-a209-1c8d40defcf1-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.556626 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.556604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.556727 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.556710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.564230 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.564199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhnc\" (UniqueName: \"kubernetes.io/projected/b315ab5e-5d34-4c17-be7f-777549b6ce3a-kube-api-access-mbhnc\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:27.564230 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.564227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzq6x\" (UniqueName: \"kubernetes.io/projected/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-api-access-fzq6x\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:27.799454 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.799416    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe3aaec4-49c4-4877-bc18-c7a48dc8d795" path="/var/lib/kubelet/pods/fe3aaec4-49c4-4877-bc18-c7a48dc8d795/volumes"
Jul 20 09:58:27.957862 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.957831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:27.960056 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:27.960037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60023925-65ad-470d-aa92-bfe998e9e857-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-24hlw\" (UID: \"60023925-65ad-470d-aa92-bfe998e9e857\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:28.059037 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.058946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-tls\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:28.059037 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.058988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:28.061375 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.061347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b315ab5e-5d34-4c17-be7f-777549b6ce3a-node-exporter-tls\") pod \"node-exporter-nqbpx\" (UID: \"b315ab5e-5d34-4c17-be7f-777549b6ce3a\") " pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:28.061487 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.061469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/731c943c-9af0-4cdd-a209-1c8d40defcf1-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jlgj7\" (UID: \"731c943c-9af0-4cdd-a209-1c8d40defcf1\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:28.247338 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.247303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"
Jul 20 09:58:28.258893 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.258868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"
Jul 20 09:58:28.274422 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.274395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nqbpx"
Jul 20 09:58:28.288170 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:28.288129    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb315ab5e_5d34_4c17_be7f_777549b6ce3a.slice/crio-370ffb55a43f9275a0f355a074007303349c7dd99e83a9e63c144f80cb9a570d WatchSource:0}: Error finding container 370ffb55a43f9275a0f355a074007303349c7dd99e83a9e63c144f80cb9a570d: Status 404 returned error can't find the container with id 370ffb55a43f9275a0f355a074007303349c7dd99e83a9e63c144f80cb9a570d
Jul 20 09:58:28.403978 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.403936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw"]
Jul 20 09:58:28.409093 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:28.409068    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60023925_65ad_470d_aa92_bfe998e9e857.slice/crio-c03feab815f46a8e6ad541e0a80f0e0c39b429f42f42c679802b4169f3e16e64 WatchSource:0}: Error finding container c03feab815f46a8e6ad541e0a80f0e0c39b429f42f42c679802b4169f3e16e64: Status 404 returned error can't find the container with id c03feab815f46a8e6ad541e0a80f0e0c39b429f42f42c679802b4169f3e16e64
Jul 20 09:58:28.425925 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.425899    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jlgj7"]
Jul 20 09:58:28.428984 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:28.428960    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731c943c_9af0_4cdd_a209_1c8d40defcf1.slice/crio-66dc89698cc6225c51f9d9346e3bea361c0308e1233d9314dce54e411777a3f7 WatchSource:0}: Error finding container 66dc89698cc6225c51f9d9346e3bea361c0308e1233d9314dce54e411777a3f7: Status 404 returned error can't find the container with id 66dc89698cc6225c51f9d9346e3bea361c0308e1233d9314dce54e411777a3f7
Jul 20 09:58:28.478073 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.478047    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:58:28.483054 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.482838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.485308 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.485263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 20 09:58:28.485545 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.485517    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gt2dh\""
Jul 20 09:58:28.485652 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.485633    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 20 09:58:28.485846 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.485826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 20 09:58:28.485977 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.485958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 20 09:58:28.486034 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.485990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 20 09:58:28.486034 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.486006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 20 09:58:28.486149 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.485958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 20 09:58:28.486207 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.486180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 20 09:58:28.486387 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.486369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 20 09:58:28.503836 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.503810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzx5\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-kube-api-access-glzx5\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-out\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-web-config\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.564051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.563896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-web-config\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.664866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.665115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.665115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzx5\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-kube-api-access-glzx5\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.665115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.665115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-out\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.665115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.664982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.665461 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.665440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.667331 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.666682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.667331 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.667264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.668606 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.668507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.668893 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.668848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-web-config\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.669048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.669018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.669137 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.669023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.669776 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.669752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.669776 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.669771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-out\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.670470 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.670411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.671828 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.671807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.673167 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.673147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzx5\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-kube-api-access-glzx5\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.673902 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.673882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.806465 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.806428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:58:28.985105 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:28.985001    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:58:28.992174 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:28.992141    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7e8727_5929_4b94_bee8_25197c3e4f37.slice/crio-c783d3723064b0f413e6b86708032f0ee2dca6d84f6ffe712dc010f76f9d21f7 WatchSource:0}: Error finding container c783d3723064b0f413e6b86708032f0ee2dca6d84f6ffe712dc010f76f9d21f7: Status 404 returned error can't find the container with id c783d3723064b0f413e6b86708032f0ee2dca6d84f6ffe712dc010f76f9d21f7
Jul 20 09:58:29.281789 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.281753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqbpx" event={"ID":"b315ab5e-5d34-4c17-be7f-777549b6ce3a","Type":"ContainerStarted","Data":"370ffb55a43f9275a0f355a074007303349c7dd99e83a9e63c144f80cb9a570d"}
Jul 20 09:58:29.283646 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.283616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw" event={"ID":"60023925-65ad-470d-aa92-bfe998e9e857","Type":"ContainerStarted","Data":"09fc2e266d0fdcc0358db21219ee1c200dae8f550d6805d27387204664649a69"}
Jul 20 09:58:29.283770 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.283674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw" event={"ID":"60023925-65ad-470d-aa92-bfe998e9e857","Type":"ContainerStarted","Data":"ab4fb9f0782858b1228dcd0c3709138c4a79ec27517e3b381fbc4ed75598a1c0"}
Jul 20 09:58:29.283770 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.283703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw" event={"ID":"60023925-65ad-470d-aa92-bfe998e9e857","Type":"ContainerStarted","Data":"c03feab815f46a8e6ad541e0a80f0e0c39b429f42f42c679802b4169f3e16e64"}
Jul 20 09:58:29.284771 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.284740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerStarted","Data":"c783d3723064b0f413e6b86708032f0ee2dca6d84f6ffe712dc010f76f9d21f7"}
Jul 20 09:58:29.285916 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.285894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7" event={"ID":"731c943c-9af0-4cdd-a209-1c8d40defcf1","Type":"ContainerStarted","Data":"66dc89698cc6225c51f9d9346e3bea361c0308e1233d9314dce54e411777a3f7"}
Jul 20 09:58:29.720811 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.720772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:29.721259 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.720830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:29.726691 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:29.726669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:30.294339 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.293703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7" event={"ID":"731c943c-9af0-4cdd-a209-1c8d40defcf1","Type":"ContainerStarted","Data":"ab4ff254497688cbd0c9a88e85eb9e8aaf55f3ac6279612a35652bcf19c56314"}
Jul 20 09:58:30.294339 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.293782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7" event={"ID":"731c943c-9af0-4cdd-a209-1c8d40defcf1","Type":"ContainerStarted","Data":"19636f7bef50ce5224a8b3d3391a38f40ed054c41b02b3842b815267b6f37737"}
Jul 20 09:58:30.294339 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.293815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7" event={"ID":"731c943c-9af0-4cdd-a209-1c8d40defcf1","Type":"ContainerStarted","Data":"c1dcf873cc295746bf0d448d2cf804935fa952de9b04b4abff29197775368f68"}
Jul 20 09:58:30.296144 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.296111    2577 generic.go:356] "Generic (PLEG): container finished" podID="b315ab5e-5d34-4c17-be7f-777549b6ce3a" containerID="a8e2b250c027cb9c655e01a285af91f6e1e6ad4c04f9bda7dd5b0c4d3fc31011" exitCode=0
Jul 20 09:58:30.296267 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.296197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqbpx" event={"ID":"b315ab5e-5d34-4c17-be7f-777549b6ce3a","Type":"ContainerDied","Data":"a8e2b250c027cb9c655e01a285af91f6e1e6ad4c04f9bda7dd5b0c4d3fc31011"}
Jul 20 09:58:30.299756 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.299720    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw" event={"ID":"60023925-65ad-470d-aa92-bfe998e9e857","Type":"ContainerStarted","Data":"0bfb4a8b0252d9316f5c5aa1b241145eda4b4ac3190d754cc5edcfc37bf120a4"}
Jul 20 09:58:30.307319 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.307294    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:58:30.315000 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.314948    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-jlgj7" podStartSLOduration=1.859931032 podStartE2EDuration="3.314932819s" totalImagesPullingTime="1.455001787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:30.314457458 +0000 UTC m=+120.946723507" watchObservedRunningTime="2026-07-20 09:58:30.314932819 +0000 UTC m=+120.947198866"
Jul 20 09:58:30.371885 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.371642    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-24hlw" podStartSLOduration=2.028524888 podStartE2EDuration="3.37161853s" totalImagesPullingTime="1.343093642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:30.370025385 +0000 UTC m=+121.002291435" watchObservedRunningTime="2026-07-20 09:58:30.37161853 +0000 UTC m=+121.003884580"
Jul 20 09:58:30.389659 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.389319    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59497659dd-46q5x"]
Jul 20 09:58:30.501759 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.501723    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6597b86945-qg9rk"]
Jul 20 09:58:30.506353 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.506315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.508855 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.508690    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 20 09:58:30.508855 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.508716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5cqe70gi6pmam\""
Jul 20 09:58:30.508855 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.508729    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 20 09:58:30.508855 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.508690    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 20 09:58:30.509313 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.509160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 20 09:58:30.509313 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.509187    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r6wc5\""
Jul 20 09:58:30.509313 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.509209    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 20 09:58:30.522621 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.522595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6597b86945-qg9rk"]
Jul 20 09:58:30.586021 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.585935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.586021 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.585974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-tls\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.586239 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.586042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwq9\" (UniqueName: \"kubernetes.io/projected/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-kube-api-access-scwq9\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.586239 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.586123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.586239 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.586147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.586239 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.586181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-metrics-client-ca\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.586239 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.586202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-grpc-tls\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.586239 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.586223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687443 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwq9\" (UniqueName: \"kubernetes.io/projected/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-kube-api-access-scwq9\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687554 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687554 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687652 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-metrics-client-ca\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687652 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-grpc-tls\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687652 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687805 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.687805 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.687722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-tls\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.688808 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.688781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-metrics-client-ca\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.690772 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.690711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.690862 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.690779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.690957 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.690937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-grpc-tls\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.691006 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.690955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.691183 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.691167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.691220 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.691172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-secret-thanos-querier-tls\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.694902 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.694864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwq9\" (UniqueName: \"kubernetes.io/projected/0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294-kube-api-access-scwq9\") pod \"thanos-querier-6597b86945-qg9rk\" (UID: \"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294\") " pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.819017 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.818968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:30.955167 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:30.955138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6597b86945-qg9rk"]
Jul 20 09:58:30.960595 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:30.960566    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea8561e_a9a8_4e77_99ef_5a6ae6c3e294.slice/crio-623757f605e83c55720ec2e49dd123ebbb819d1b1c816ab0abc2704762df794d WatchSource:0}: Error finding container 623757f605e83c55720ec2e49dd123ebbb819d1b1c816ab0abc2704762df794d: Status 404 returned error can't find the container with id 623757f605e83c55720ec2e49dd123ebbb819d1b1c816ab0abc2704762df794d
Jul 20 09:58:31.306373 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.306330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqbpx" event={"ID":"b315ab5e-5d34-4c17-be7f-777549b6ce3a","Type":"ContainerStarted","Data":"abcfdecea1d9e586ae51b53e315d7ded22e3311ce88288bfa1e037cc9c7db62a"}
Jul 20 09:58:31.306570 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.306406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nqbpx" event={"ID":"b315ab5e-5d34-4c17-be7f-777549b6ce3a","Type":"ContainerStarted","Data":"20db6dc3c95dc2bdfaa54b2e79f382286e1ddcd05cd877ce1f7977df6d39321e"}
Jul 20 09:58:31.307590 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.307562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" event={"ID":"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294","Type":"ContainerStarted","Data":"623757f605e83c55720ec2e49dd123ebbb819d1b1c816ab0abc2704762df794d"}
Jul 20 09:58:31.309105 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.309080    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerID="4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77" exitCode=0
Jul 20 09:58:31.309205 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.309120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77"}
Jul 20 09:58:31.326980 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.326919    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nqbpx" podStartSLOduration=2.61167332 podStartE2EDuration="4.326896001s" totalImagesPullingTime="1.592113371s" totalInitContainerRuntime="123.10931ms" isStatefulPod=true podCreationTimestamp="2026-07-20 09:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:31.32588957 +0000 UTC m=+121.958155619" watchObservedRunningTime="2026-07-20 09:58:31.326896001 +0000 UTC m=+121.959162051"
Jul 20 09:58:31.631680 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.631591    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76c658bcb6-46bmb"]
Jul 20 09:58:31.635171 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.635150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.637924 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.637578    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 20 09:58:31.637924 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.637763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nbvls\""
Jul 20 09:58:31.637924 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.637768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-20ugst3tu78pe\""
Jul 20 09:58:31.637924 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.637825    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 20 09:58:31.637924 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.637884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 20 09:58:31.638240 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.638015    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 20 09:58:31.643326 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.643302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76c658bcb6-46bmb"]
Jul 20 09:58:31.698422 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.698391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-client-ca-bundle\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.698625 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.698435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-secret-metrics-server-client-certs\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.698625 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.698462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62209b79-67dd-4b97-ade3-0b3c017141d7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.698625 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.698571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62209b79-67dd-4b97-ade3-0b3c017141d7-audit-log\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.698625 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.698612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpggz\" (UniqueName: \"kubernetes.io/projected/62209b79-67dd-4b97-ade3-0b3c017141d7-kube-api-access-dpggz\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.698841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.698663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62209b79-67dd-4b97-ade3-0b3c017141d7-metrics-server-audit-profiles\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.698841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.698722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-secret-metrics-server-tls\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804209 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62209b79-67dd-4b97-ade3-0b3c017141d7-audit-log\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804369 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpggz\" (UniqueName: \"kubernetes.io/projected/62209b79-67dd-4b97-ade3-0b3c017141d7-kube-api-access-dpggz\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804369 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62209b79-67dd-4b97-ade3-0b3c017141d7-metrics-server-audit-profiles\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804369 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-secret-metrics-server-tls\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804566 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-client-ca-bundle\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804566 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-secret-metrics-server-client-certs\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804566 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62209b79-67dd-4b97-ade3-0b3c017141d7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.804718 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.804623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62209b79-67dd-4b97-ade3-0b3c017141d7-audit-log\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.805206 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.805181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62209b79-67dd-4b97-ade3-0b3c017141d7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.805325 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.805258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62209b79-67dd-4b97-ade3-0b3c017141d7-metrics-server-audit-profiles\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.807351 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.807328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-secret-metrics-server-tls\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.807938 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.807917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-secret-metrics-server-client-certs\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.808067 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.807986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62209b79-67dd-4b97-ade3-0b3c017141d7-client-ca-bundle\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.812715 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.812696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpggz\" (UniqueName: \"kubernetes.io/projected/62209b79-67dd-4b97-ade3-0b3c017141d7-kube-api-access-dpggz\") pod \"metrics-server-76c658bcb6-46bmb\" (UID: \"62209b79-67dd-4b97-ade3-0b3c017141d7\") " pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:31.947935 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:31.947840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:32.122827 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.121330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"]
Jul 20 09:58:32.126274 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.126247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:32.129106 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.129081    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 20 09:58:32.129210 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.129086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dqq59\""
Jul 20 09:58:32.136063 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.136040    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"]
Jul 20 09:58:32.156585 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.156562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76c658bcb6-46bmb"]
Jul 20 09:58:32.158089 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:32.158061    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62209b79_67dd_4b97_ade3_0b3c017141d7.slice/crio-978510f2f1ecd5b5564f13ad5908d32b2804ee2ba16ae37b5d43d951221a540a WatchSource:0}: Error finding container 978510f2f1ecd5b5564f13ad5908d32b2804ee2ba16ae37b5d43d951221a540a: Status 404 returned error can't find the container with id 978510f2f1ecd5b5564f13ad5908d32b2804ee2ba16ae37b5d43d951221a540a
Jul 20 09:58:32.208662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.208632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b3ce2356-5a10-4a03-afb1-98ce88d4f4fe-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-97vtd\" (UID: \"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:32.208876 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.208752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fmq\" (UniqueName: \"kubernetes.io/projected/b3ce2356-5a10-4a03-afb1-98ce88d4f4fe-kube-api-access-r6fmq\") pod \"monitoring-plugin-7b7b7bc8bf-97vtd\" (UID: \"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:32.309352 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.309322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fmq\" (UniqueName: \"kubernetes.io/projected/b3ce2356-5a10-4a03-afb1-98ce88d4f4fe-kube-api-access-r6fmq\") pod \"monitoring-plugin-7b7b7bc8bf-97vtd\" (UID: \"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:32.309505 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.309411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b3ce2356-5a10-4a03-afb1-98ce88d4f4fe-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-97vtd\" (UID: \"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:32.312460 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.312436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b3ce2356-5a10-4a03-afb1-98ce88d4f4fe-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-97vtd\" (UID: \"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:32.315107 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.315081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb" event={"ID":"62209b79-67dd-4b97-ade3-0b3c017141d7","Type":"ContainerStarted","Data":"978510f2f1ecd5b5564f13ad5908d32b2804ee2ba16ae37b5d43d951221a540a"}
Jul 20 09:58:32.322940 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.322916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fmq\" (UniqueName: \"kubernetes.io/projected/b3ce2356-5a10-4a03-afb1-98ce88d4f4fe-kube-api-access-r6fmq\") pod \"monitoring-plugin-7b7b7bc8bf-97vtd\" (UID: \"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:32.437877 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:32.437838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:33.327667 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.327569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerStarted","Data":"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"}
Jul 20 09:58:33.330932 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.330506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" event={"ID":"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294","Type":"ContainerStarted","Data":"1ceaa420c1080879da901c899e1a28a4ffbdc226747392ed95d9b0234c796eee"}
Jul 20 09:58:33.388348 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.388279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"]
Jul 20 09:58:33.394418 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:33.394389    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ce2356_5a10_4a03_afb1_98ce88d4f4fe.slice/crio-29b99b555973185bf942f50881832d52896be75b03a6e1f4f2b3d133600be05b WatchSource:0}: Error finding container 29b99b555973185bf942f50881832d52896be75b03a6e1f4f2b3d133600be05b: Status 404 returned error can't find the container with id 29b99b555973185bf942f50881832d52896be75b03a6e1f4f2b3d133600be05b
Jul 20 09:58:33.681365 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.681324    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:58:33.685587 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.685559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.687796 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.687770    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 20 09:58:33.693658 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.693634    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 20 09:58:33.693982 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.693958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 20 09:58:33.694122 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.694101    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 20 09:58:33.694233 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.693963    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 20 09:58:33.694277 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.694254    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-89s2sppcqh6up\""
Jul 20 09:58:33.694498 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.694477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 20 09:58:33.694623 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.694608    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 20 09:58:33.694753 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.694734    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 20 09:58:33.695598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.694992    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w7lvv\""
Jul 20 09:58:33.695598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.695147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 20 09:58:33.695598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.695208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 20 09:58:33.695598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.695243    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 20 09:58:33.695598 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.695344    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 20 09:58:33.696272 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.696251    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 20 09:58:33.698908 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.698885    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 20 09:58:33.710499 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.710464    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:58:33.722394 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722394 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722394 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-config-out\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjr9\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-kube-api-access-jmjr9\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-config\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-web-config\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.722866 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.723457 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.722935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.723457 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.723002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824159 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824159 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-config-out\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjr9\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-kube-api-access-jmjr9\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824412 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-config\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-web-config\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.824746 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.825187 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.825187 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.824836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.825289 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.825198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.825690 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.825661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.826886 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.826566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.826886 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.826592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.827986 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.827683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.828094 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.827978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.829139 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.828779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-config-out\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.829139 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.828879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.829310 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.829211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.829924 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.829802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.829924 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.829802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.830726 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.830625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.831293 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.831253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-config\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.831943 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.831555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.831943 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.831903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.832083 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.831984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.833113 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.833087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.833349 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.833325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-web-config\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.834732 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.834710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:33.835437 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:33.835415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjr9\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-kube-api-access-jmjr9\") pod \"prometheus-k8s-0\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:34.002500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.002199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:34.339316 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.339181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerStarted","Data":"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"}
Jul 20 09:58:34.339316 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.339273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerStarted","Data":"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"}
Jul 20 09:58:34.339316 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.339316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerStarted","Data":"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"}
Jul 20 09:58:34.339844 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.339357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerStarted","Data":"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"}
Jul 20 09:58:34.340744 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.340706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd" event={"ID":"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe","Type":"ContainerStarted","Data":"29b99b555973185bf942f50881832d52896be75b03a6e1f4f2b3d133600be05b"}
Jul 20 09:58:34.345061 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.343876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" event={"ID":"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294","Type":"ContainerStarted","Data":"de2bbdaacf9ea72bd78a4ce1591f08be22fd0d8b39ff574aa7dbd2e177594b25"}
Jul 20 09:58:34.345061 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.343956    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" event={"ID":"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294","Type":"ContainerStarted","Data":"cad48dd5dbc519dea44277317cb89597ae8221719cff5ed7ee3d5587b406386d"}
Jul 20 09:58:34.461070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:34.461034    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:58:34.467119 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:34.467080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf003670_74f2_490e_911d_dced2d7b4da1.slice/crio-40d2a9870ecfc5d087dcacd8d89036998206b314a51626b89541476fb095508f WatchSource:0}: Error finding container 40d2a9870ecfc5d087dcacd8d89036998206b314a51626b89541476fb095508f: Status 404 returned error can't find the container with id 40d2a9870ecfc5d087dcacd8d89036998206b314a51626b89541476fb095508f
Jul 20 09:58:35.351986 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.351945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" event={"ID":"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294","Type":"ContainerStarted","Data":"efd7f59381aa31e05561e8822b99e0af850e33073e9e107586cc3427b2cb7890"}
Jul 20 09:58:35.352467 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.352066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" event={"ID":"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294","Type":"ContainerStarted","Data":"0b422ed4926faa3c1f774ac601e2511fb52a9bc24f8c9ebe1c784e119a3aec76"}
Jul 20 09:58:35.353918 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.353572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb" event={"ID":"62209b79-67dd-4b97-ade3-0b3c017141d7","Type":"ContainerStarted","Data":"b4dc842cc647c12e1f6fe12ed07a08ec7a3258bbf9bc172e5b81ac5882456fef"}
Jul 20 09:58:35.358674 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.358647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerStarted","Data":"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"}
Jul 20 09:58:35.360700 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.360671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd" event={"ID":"b3ce2356-5a10-4a03-afb1-98ce88d4f4fe","Type":"ContainerStarted","Data":"628af9eadfc52938124541ce7bdac54e56e4394e6303096cec13218e2d233750"}
Jul 20 09:58:35.360860 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.360840    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:35.362661 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.362636    2577 generic.go:356] "Generic (PLEG): container finished" podID="af003670-74f2-490e-911d-dced2d7b4da1" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa" exitCode=0
Jul 20 09:58:35.362785 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.362682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"}
Jul 20 09:58:35.362785 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.362740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerStarted","Data":"40d2a9870ecfc5d087dcacd8d89036998206b314a51626b89541476fb095508f"}
Jul 20 09:58:35.366048 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.366030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd"
Jul 20 09:58:35.383140 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.383065    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb" podStartSLOduration=2.19314433 podStartE2EDuration="4.383042351s" totalImagesPullingTime="2.189898021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:35.382492524 +0000 UTC m=+126.014758573" watchObservedRunningTime="2026-07-20 09:58:35.383042351 +0000 UTC m=+126.015308405"
Jul 20 09:58:35.426455 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.426344    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.360755694 podStartE2EDuration="7.426322125s" totalImagesPullingTime="5.050688238s" totalInitContainerRuntime="14.878193ms" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:35.42385705 +0000 UTC m=+126.056123100" watchObservedRunningTime="2026-07-20 09:58:35.426322125 +0000 UTC m=+126.058588172"
Jul 20 09:58:35.505317 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:35.505261    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-97vtd" podStartSLOduration=1.811857635 podStartE2EDuration="3.505245428s" totalImagesPullingTime="1.693387793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:35.504021198 +0000 UTC m=+126.136287248" watchObservedRunningTime="2026-07-20 09:58:35.505245428 +0000 UTC m=+126.137511544"
Jul 20 09:58:36.371049 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:36.371011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" event={"ID":"0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294","Type":"ContainerStarted","Data":"76f0fd666a3a0e53f07836c60e7c379cf1d9bd45642e29503d956b2f76b0a777"}
Jul 20 09:58:36.400408 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:36.400352    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk" podStartSLOduration=2.52463476 podStartE2EDuration="6.400328284s" totalImagesPullingTime="3.875693524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:36.39746962 +0000 UTC m=+127.029735680" watchObservedRunningTime="2026-07-20 09:58:36.400328284 +0000 UTC m=+127.032594333"
Jul 20 09:58:37.375810 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:37.375678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:38.386466 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:38.386446    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6597b86945-qg9rk"
Jul 20 09:58:39.387492 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.387454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerStarted","Data":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"}
Jul 20 09:58:39.387921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.387522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerStarted","Data":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"}
Jul 20 09:58:39.387921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.387575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerStarted","Data":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"}
Jul 20 09:58:39.387921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.387601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerStarted","Data":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"}
Jul 20 09:58:39.387921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.387641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerStarted","Data":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"}
Jul 20 09:58:39.387921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.387666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerStarted","Data":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"}
Jul 20 09:58:39.444964 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.444911    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.439057594 podStartE2EDuration="6.444893292s" totalImagesPullingTime="2.971228777s" totalInitContainerRuntime="34.606921ms" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:39.443470569 +0000 UTC m=+130.075736654" watchObservedRunningTime="2026-07-20 09:58:39.444893292 +0000 UTC m=+130.077159740"
Jul 20 09:58:39.589016 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.588972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:58:39.591379 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.591351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7da1e17e-2528-47b6-90a5-cc7f4d4103a8-metrics-certs\") pod \"network-metrics-daemon-mg256\" (UID: \"7da1e17e-2528-47b6-90a5-cc7f4d4103a8\") " pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:58:39.806263 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.806234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x8bv\""
Jul 20 09:58:39.813967 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.813944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mg256"
Jul 20 09:58:39.978993 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:39.978964    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mg256"]
Jul 20 09:58:39.979489 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:58:39.979466    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da1e17e_2528_47b6_90a5_cc7f4d4103a8.slice/crio-7938234e590667d1d95f112e121a7d2049cf594a7bd1e52d41995b06bf70048d WatchSource:0}: Error finding container 7938234e590667d1d95f112e121a7d2049cf594a7bd1e52d41995b06bf70048d: Status 404 returned error can't find the container with id 7938234e590667d1d95f112e121a7d2049cf594a7bd1e52d41995b06bf70048d
Jul 20 09:58:40.393139 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:40.393093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mg256" event={"ID":"7da1e17e-2528-47b6-90a5-cc7f4d4103a8","Type":"ContainerStarted","Data":"7938234e590667d1d95f112e121a7d2049cf594a7bd1e52d41995b06bf70048d"}
Jul 20 09:58:42.405676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:42.405643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mg256" event={"ID":"7da1e17e-2528-47b6-90a5-cc7f4d4103a8","Type":"ContainerStarted","Data":"7785e39f25a1ab74fbcfff7507b74703a272eb7a259908f416ae24c622fbfb8b"}
Jul 20 09:58:42.406026 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:42.405694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mg256" event={"ID":"7da1e17e-2528-47b6-90a5-cc7f4d4103a8","Type":"ContainerStarted","Data":"a6a137a297581fb94106cae4d7cebac4470274db2ea48f2b8069c16c355c1750"}
Jul 20 09:58:42.427920 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:42.427874    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mg256" podStartSLOduration=131.029678234 podStartE2EDuration="2m12.427858702s" totalImagesPullingTime="1.398180468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:42.424994618 +0000 UTC m=+133.057260666" watchObservedRunningTime="2026-07-20 09:58:42.427858702 +0000 UTC m=+133.060124750"
Jul 20 09:58:44.002826 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:44.002750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:58:44.145334 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:44.145306    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7978db7485-jmw9s"]
Jul 20 09:58:51.948411 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:51.948376    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:51.948888 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:51.948433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:58:55.415213 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.415148    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59497659dd-46q5x" podUID="14b5e2e7-b679-4833-bb68-7c7ee01a135e" containerName="console" containerID="cri-o://99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8" gracePeriod=15
Jul 20 09:58:55.660159 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.660131    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:55.843696 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.843663    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-config\") pod \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") "
Jul 20 09:58:55.843696 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.843704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndrp4\" (UniqueName: \"kubernetes.io/projected/14b5e2e7-b679-4833-bb68-7c7ee01a135e-kube-api-access-ndrp4\") pod \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") "
Jul 20 09:58:55.843933 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.843748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-serving-cert\") pod \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") "
Jul 20 09:58:55.843933 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.843789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-service-ca\") pod \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") "
Jul 20 09:58:55.843933 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.843839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-oauth-serving-cert\") pod \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") "
Jul 20 09:58:55.843933 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.843862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-oauth-config\") pod \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\" (UID: \"14b5e2e7-b679-4833-bb68-7c7ee01a135e\") "
Jul 20 09:58:55.844180 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.844140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-config" (OuterVolumeSpecName: "console-config") pod "14b5e2e7-b679-4833-bb68-7c7ee01a135e" (UID: "14b5e2e7-b679-4833-bb68-7c7ee01a135e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:58:55.844243 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.844222    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-service-ca" (OuterVolumeSpecName: "service-ca") pod "14b5e2e7-b679-4833-bb68-7c7ee01a135e" (UID: "14b5e2e7-b679-4833-bb68-7c7ee01a135e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:58:55.844243 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.844218    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "14b5e2e7-b679-4833-bb68-7c7ee01a135e" (UID: "14b5e2e7-b679-4833-bb68-7c7ee01a135e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:58:55.846207 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.846175    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "14b5e2e7-b679-4833-bb68-7c7ee01a135e" (UID: "14b5e2e7-b679-4833-bb68-7c7ee01a135e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:58:55.846329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.846217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "14b5e2e7-b679-4833-bb68-7c7ee01a135e" (UID: "14b5e2e7-b679-4833-bb68-7c7ee01a135e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:58:55.846329 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.846262    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b5e2e7-b679-4833-bb68-7c7ee01a135e-kube-api-access-ndrp4" (OuterVolumeSpecName: "kube-api-access-ndrp4") pod "14b5e2e7-b679-4833-bb68-7c7ee01a135e" (UID: "14b5e2e7-b679-4833-bb68-7c7ee01a135e"). InnerVolumeSpecName "kube-api-access-ndrp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:58:55.944810 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.944779    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-service-ca\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:55.944810 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.944804    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-oauth-serving-cert\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:55.944810 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.944814    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-oauth-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:55.945038 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.944823    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:55.945038 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.944831    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrp4\" (UniqueName: \"kubernetes.io/projected/14b5e2e7-b679-4833-bb68-7c7ee01a135e-kube-api-access-ndrp4\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:55.945038 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:55.944841    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b5e2e7-b679-4833-bb68-7c7ee01a135e-console-serving-cert\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:58:56.471687 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.471651    2577 generic.go:356] "Generic (PLEG): container finished" podID="14b5e2e7-b679-4833-bb68-7c7ee01a135e" containerID="99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8" exitCode=0
Jul 20 09:58:56.472143 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.471708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59497659dd-46q5x"
Jul 20 09:58:56.472143 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.471747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59497659dd-46q5x" event={"ID":"14b5e2e7-b679-4833-bb68-7c7ee01a135e","Type":"ContainerDied","Data":"99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8"}
Jul 20 09:58:56.472143 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.471811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59497659dd-46q5x" event={"ID":"14b5e2e7-b679-4833-bb68-7c7ee01a135e","Type":"ContainerDied","Data":"01d9df678f285c6ca072dee3c71e27a938030702ea7501ab458fba6da5824026"}
Jul 20 09:58:56.472143 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.471835    2577 scope.go:117] "RemoveContainer" containerID="99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8"
Jul 20 09:58:56.481382 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.481366    2577 scope.go:117] "RemoveContainer" containerID="99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8"
Jul 20 09:58:56.481665 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:58:56.481647    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8\": container with ID starting with 99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8 not found: ID does not exist" containerID="99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8"
Jul 20 09:58:56.481721 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.481674    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8"} err="failed to get container status \"99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8\": rpc error: code = NotFound desc = could not find container \"99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8\": container with ID starting with 99de0c474064d762a566c882fdf552a601e2996e8f03c8e7970d3dafd06223b8 not found: ID does not exist"
Jul 20 09:58:56.492461 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.492435    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59497659dd-46q5x"]
Jul 20 09:58:56.496627 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:56.496604    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59497659dd-46q5x"]
Jul 20 09:58:57.795368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:57.795333    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b5e2e7-b679-4833-bb68-7c7ee01a135e" path="/var/lib/kubelet/pods/14b5e2e7-b679-4833-bb68-7c7ee01a135e/volumes"
Jul 20 09:58:59.486889 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:59.486851    2577 generic.go:356] "Generic (PLEG): container finished" podID="0139a14f-e8e2-47c4-aa1c-8eece655bde0" containerID="bbacb4fdd451f4bb29b00ac23f91e48bfaa44cb400b5f2587f8d3199fed9268c" exitCode=0
Jul 20 09:58:59.487366 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:59.486929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd" event={"ID":"0139a14f-e8e2-47c4-aa1c-8eece655bde0","Type":"ContainerDied","Data":"bbacb4fdd451f4bb29b00ac23f91e48bfaa44cb400b5f2587f8d3199fed9268c"}
Jul 20 09:58:59.487366 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:58:59.487338    2577 scope.go:117] "RemoveContainer" containerID="bbacb4fdd451f4bb29b00ac23f91e48bfaa44cb400b5f2587f8d3199fed9268c"
Jul 20 09:59:00.492784 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:00.492752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-dmmwd" event={"ID":"0139a14f-e8e2-47c4-aa1c-8eece655bde0","Type":"ContainerStarted","Data":"b6fcc3dfae1ed88ace7dc00046c3cc3827c90bd4f449628d72fb0b8c05eab127"}
Jul 20 09:59:09.166576 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.166450    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7978db7485-jmw9s" podUID="32d0e7fa-ab8e-4cab-8378-0536e8b40032" containerName="console" containerID="cri-o://363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4" gracePeriod=15
Jul 20 09:59:09.418561 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.418484    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:59:09.462500 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-trusted-ca-bundle\") pod \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") "
Jul 20 09:59:09.462669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462519    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-serving-cert\") pod \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") "
Jul 20 09:59:09.462669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh2hg\" (UniqueName: \"kubernetes.io/projected/32d0e7fa-ab8e-4cab-8378-0536e8b40032-kube-api-access-fh2hg\") pod \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") "
Jul 20 09:59:09.462669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-service-ca\") pod \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") "
Jul 20 09:59:09.462669 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-config\") pod \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") "
Jul 20 09:59:09.462855 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-oauth-config\") pod \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") "
Jul 20 09:59:09.462855 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462747    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-oauth-serving-cert\") pod \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\" (UID: \"32d0e7fa-ab8e-4cab-8378-0536e8b40032\") "
Jul 20 09:59:09.462962 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "32d0e7fa-ab8e-4cab-8378-0536e8b40032" (UID: "32d0e7fa-ab8e-4cab-8378-0536e8b40032"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:09.463019 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.462993    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-service-ca" (OuterVolumeSpecName: "service-ca") pod "32d0e7fa-ab8e-4cab-8378-0536e8b40032" (UID: "32d0e7fa-ab8e-4cab-8378-0536e8b40032"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:09.463207 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.463176    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-trusted-ca-bundle\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:09.463207 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.463181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "32d0e7fa-ab8e-4cab-8378-0536e8b40032" (UID: "32d0e7fa-ab8e-4cab-8378-0536e8b40032"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:09.463801 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.463770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-config" (OuterVolumeSpecName: "console-config") pod "32d0e7fa-ab8e-4cab-8378-0536e8b40032" (UID: "32d0e7fa-ab8e-4cab-8378-0536e8b40032"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:09.464952 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.464921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "32d0e7fa-ab8e-4cab-8378-0536e8b40032" (UID: "32d0e7fa-ab8e-4cab-8378-0536e8b40032"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:09.465041 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.464951    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d0e7fa-ab8e-4cab-8378-0536e8b40032-kube-api-access-fh2hg" (OuterVolumeSpecName: "kube-api-access-fh2hg") pod "32d0e7fa-ab8e-4cab-8378-0536e8b40032" (UID: "32d0e7fa-ab8e-4cab-8378-0536e8b40032"). InnerVolumeSpecName "kube-api-access-fh2hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:09.465041 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.464947    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "32d0e7fa-ab8e-4cab-8378-0536e8b40032" (UID: "32d0e7fa-ab8e-4cab-8378-0536e8b40032"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:09.531280 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.531247    2577 generic.go:356] "Generic (PLEG): container finished" podID="32d0e7fa-ab8e-4cab-8378-0536e8b40032" containerID="363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4" exitCode=0
Jul 20 09:59:09.531409 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.531298    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7978db7485-jmw9s"
Jul 20 09:59:09.531409 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.531337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7978db7485-jmw9s" event={"ID":"32d0e7fa-ab8e-4cab-8378-0536e8b40032","Type":"ContainerDied","Data":"363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4"}
Jul 20 09:59:09.531478 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.531410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7978db7485-jmw9s" event={"ID":"32d0e7fa-ab8e-4cab-8378-0536e8b40032","Type":"ContainerDied","Data":"9f1cd72c092b8ff7250ca0dcc1f558b4328c410ac9fd032688d01dadf76f02c4"}
Jul 20 09:59:09.531478 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.531430    2577 scope.go:117] "RemoveContainer" containerID="363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4"
Jul 20 09:59:09.540299 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.540284    2577 scope.go:117] "RemoveContainer" containerID="363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4"
Jul 20 09:59:09.540581 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:09.540561    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4\": container with ID starting with 363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4 not found: ID does not exist" containerID="363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4"
Jul 20 09:59:09.540644 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.540590    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4"} err="failed to get container status \"363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4\": rpc error: code = NotFound desc = could not find container \"363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4\": container with ID starting with 363cf06293e14b8d7dd4ae8d28c4e3b7f417c117ddf2e124de560cfc930b12a4 not found: ID does not exist"
Jul 20 09:59:09.550471 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.550453    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7978db7485-jmw9s"]
Jul 20 09:59:09.553732 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.553712    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7978db7485-jmw9s"]
Jul 20 09:59:09.564305 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.564286    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-service-ca\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:09.564382 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.564308    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:09.564382 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.564317    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-oauth-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:09.564382 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.564326    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d0e7fa-ab8e-4cab-8378-0536e8b40032-oauth-serving-cert\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:09.564382 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.564337    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d0e7fa-ab8e-4cab-8378-0536e8b40032-console-serving-cert\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:09.564382 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.564348    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh2hg\" (UniqueName: \"kubernetes.io/projected/32d0e7fa-ab8e-4cab-8378-0536e8b40032-kube-api-access-fh2hg\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:09.796850 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:09.796815    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d0e7fa-ab8e-4cab-8378-0536e8b40032" path="/var/lib/kubelet/pods/32d0e7fa-ab8e-4cab-8378-0536e8b40032/volumes"
Jul 20 09:59:11.954560 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:11.954513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:59:11.960639 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:11.960550    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76c658bcb6-46bmb"
Jul 20 09:59:34.002479 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:34.002434    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:34.019054 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:34.019029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:34.669089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:34.669060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:47.745886 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:47.745854    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:47.746368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:47.746143    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="alertmanager" containerID="cri-o://de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8" gracePeriod=120
Jul 20 09:59:47.746368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:47.746201    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-metric" containerID="cri-o://8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98" gracePeriod=120
Jul 20 09:59:47.746368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:47.746258    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-web" containerID="cri-o://9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456" gracePeriod=120
Jul 20 09:59:47.746368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:47.746306    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy" containerID="cri-o://86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c" gracePeriod=120
Jul 20 09:59:47.746636 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:47.746258    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="config-reloader" containerID="cri-o://346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e" gracePeriod=120
Jul 20 09:59:47.746636 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:47.746293    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="prom-label-proxy" containerID="cri-o://51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3" gracePeriod=120
Jul 20 09:59:48.709654 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709624    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerID="51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3" exitCode=0
Jul 20 09:59:48.709654 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709649    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerID="86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c" exitCode=0
Jul 20 09:59:48.709654 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709655    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerID="346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e" exitCode=0
Jul 20 09:59:48.709654 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709660    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerID="de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8" exitCode=0
Jul 20 09:59:48.709911 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"}
Jul 20 09:59:48.709911 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"}
Jul 20 09:59:48.709911 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"}
Jul 20 09:59:48.709911 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.709818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"}
Jul 20 09:59:48.996180 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:48.996157    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.119351 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-main-db\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-metrics-client-ca\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119382    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-out\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glzx5\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-kube-api-access-glzx5\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-trusted-ca-bundle\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119480    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-volume\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119568 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119523    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-main-tls\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119895 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-cluster-tls-config\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119895 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119630    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119895 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119895 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-web-config\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119895 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-web\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.119895 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119755    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-tls-assets\") pod \"7a7e8727-5929-4b94-bee8-25197c3e4f37\" (UID: \"7a7e8727-5929-4b94-bee8-25197c3e4f37\") "
Jul 20 09:59:49.120273 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:59:49.120273 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:49.120273 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.119959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:49.122841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.122809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:49.123513 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.123061    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:49.123513 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.123207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:49.123513 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.123229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:49.123513 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.123246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-volume" (OuterVolumeSpecName: "config-volume") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:49.123513 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.123393    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-kube-api-access-glzx5" (OuterVolumeSpecName: "kube-api-access-glzx5") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "kube-api-access-glzx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:49.123920 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.123877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-out" (OuterVolumeSpecName: "config-out") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:59:49.125128 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.125101    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:49.126977 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.126938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:49.135410 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.135387    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-web-config" (OuterVolumeSpecName: "web-config") pod "7a7e8727-5929-4b94-bee8-25197c3e4f37" (UID: "7a7e8727-5929-4b94-bee8-25197c3e4f37"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:49.220824 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220794    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220824 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220823    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220834    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-web-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220843    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220852    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-tls-assets\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220860    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-main-db\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220870    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-metrics-client-ca\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220878    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-out\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220886    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glzx5\" (UniqueName: \"kubernetes.io/projected/7a7e8727-5929-4b94-bee8-25197c3e4f37-kube-api-access-glzx5\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220893    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7e8727-5929-4b94-bee8-25197c3e4f37-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220901    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-config-volume\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220909    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-secret-alertmanager-main-tls\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.220985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.220916    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a7e8727-5929-4b94-bee8-25197c3e4f37-cluster-tls-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:49.716374 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.716338    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerID="8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98" exitCode=0
Jul 20 09:59:49.716374 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.716366    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerID="9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456" exitCode=0
Jul 20 09:59:49.716704 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.716433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"}
Jul 20 09:59:49.716704 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.716464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.716704 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.716509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"}
Jul 20 09:59:49.716704 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.716579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a7e8727-5929-4b94-bee8-25197c3e4f37","Type":"ContainerDied","Data":"c783d3723064b0f413e6b86708032f0ee2dca6d84f6ffe712dc010f76f9d21f7"}
Jul 20 09:59:49.716704 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.716586    2577 scope.go:117] "RemoveContainer" containerID="51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"
Jul 20 09:59:49.724879 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.724863    2577 scope.go:117] "RemoveContainer" containerID="8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"
Jul 20 09:59:49.731911 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.731893    2577 scope.go:117] "RemoveContainer" containerID="86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"
Jul 20 09:59:49.739385 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.739366    2577 scope.go:117] "RemoveContainer" containerID="9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"
Jul 20 09:59:49.741226 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.740727    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:49.746031 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.745999    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:49.751645 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.751516    2577 scope.go:117] "RemoveContainer" containerID="346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"
Jul 20 09:59:49.759047 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.759029    2577 scope.go:117] "RemoveContainer" containerID="de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"
Jul 20 09:59:49.766958 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.766937    2577 scope.go:117] "RemoveContainer" containerID="4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77"
Jul 20 09:59:49.768786 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.768766    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:49.769122 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769107    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="alertmanager"
Jul 20 09:59:49.769122 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769122    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="alertmanager"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769134    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="init-config-reloader"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769140    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="init-config-reloader"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769148    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="prom-label-proxy"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769153    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="prom-label-proxy"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769162    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d0e7fa-ab8e-4cab-8378-0536e8b40032" containerName="console"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769167    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d0e7fa-ab8e-4cab-8378-0536e8b40032" containerName="console"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769174    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b5e2e7-b679-4833-bb68-7c7ee01a135e" containerName="console"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769179    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b5e2e7-b679-4833-bb68-7c7ee01a135e" containerName="console"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769187    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="config-reloader"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769192    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="config-reloader"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769197    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-metric"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769202    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-metric"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769209    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-web"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769214    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-web"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769222    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy"
Jul 20 09:59:49.769250 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769227    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769272    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d0e7fa-ab8e-4cab-8378-0536e8b40032" containerName="console"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769278    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="config-reloader"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769286    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b5e2e7-b679-4833-bb68-7c7ee01a135e" containerName="console"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769293    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769298    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-web"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769304    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="alertmanager"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769309    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="prom-label-proxy"
Jul 20 09:59:49.770089 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.769315    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" containerName="kube-rbac-proxy-metric"
Jul 20 09:59:49.772817 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.772791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.775450 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.775431    2577 scope.go:117] "RemoveContainer" containerID="51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"
Jul 20 09:59:49.775744 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:49.775722    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3\": container with ID starting with 51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3 not found: ID does not exist" containerID="51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"
Jul 20 09:59:49.775832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.775755    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"} err="failed to get container status \"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3\": rpc error: code = NotFound desc = could not find container \"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3\": container with ID starting with 51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3 not found: ID does not exist"
Jul 20 09:59:49.775832 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.775782    2577 scope.go:117] "RemoveContainer" containerID="8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"
Jul 20 09:59:49.776054 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:49.776028    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98\": container with ID starting with 8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98 not found: ID does not exist" containerID="8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"
Jul 20 09:59:49.776115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776061    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"} err="failed to get container status \"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98\": rpc error: code = NotFound desc = could not find container \"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98\": container with ID starting with 8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98 not found: ID does not exist"
Jul 20 09:59:49.776115 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776077    2577 scope.go:117] "RemoveContainer" containerID="86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"
Jul 20 09:59:49.776308 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:49.776291    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c\": container with ID starting with 86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c not found: ID does not exist" containerID="86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"
Jul 20 09:59:49.776378 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776311    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"} err="failed to get container status \"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c\": rpc error: code = NotFound desc = could not find container \"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c\": container with ID starting with 86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c not found: ID does not exist"
Jul 20 09:59:49.776378 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776323    2577 scope.go:117] "RemoveContainer" containerID="9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"
Jul 20 09:59:49.776661 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:49.776542    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456\": container with ID starting with 9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456 not found: ID does not exist" containerID="9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"
Jul 20 09:59:49.776661 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776576    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"} err="failed to get container status \"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456\": rpc error: code = NotFound desc = could not find container \"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456\": container with ID starting with 9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456 not found: ID does not exist"
Jul 20 09:59:49.776661 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776594    2577 scope.go:117] "RemoveContainer" containerID="346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"
Jul 20 09:59:49.776874 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:49.776855    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e\": container with ID starting with 346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e not found: ID does not exist" containerID="346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"
Jul 20 09:59:49.776937 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776876    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"} err="failed to get container status \"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e\": rpc error: code = NotFound desc = could not find container \"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e\": container with ID starting with 346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e not found: ID does not exist"
Jul 20 09:59:49.776937 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776890    2577 scope.go:117] "RemoveContainer" containerID="de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"
Jul 20 09:59:49.777040 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776978    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 20 09:59:49.777040 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.776981    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 20 09:59:49.777160 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:49.777138    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8\": container with ID starting with de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8 not found: ID does not exist" containerID="de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"
Jul 20 09:59:49.777208 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777168    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"} err="failed to get container status \"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8\": rpc error: code = NotFound desc = could not find container \"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8\": container with ID starting with de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8 not found: ID does not exist"
Jul 20 09:59:49.777208 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777188    2577 scope.go:117] "RemoveContainer" containerID="4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77"
Jul 20 09:59:49.777448 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:49.777419    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77\": container with ID starting with 4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77 not found: ID does not exist" containerID="4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77"
Jul 20 09:59:49.777561 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777455    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77"} err="failed to get container status \"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77\": rpc error: code = NotFound desc = could not find container \"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77\": container with ID starting with 4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77 not found: ID does not exist"
Jul 20 09:59:49.777561 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777478    2577 scope.go:117] "RemoveContainer" containerID="51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"
Jul 20 09:59:49.777561 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 20 09:59:49.777561 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 20 09:59:49.777561 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gt2dh\""
Jul 20 09:59:49.777561 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 20 09:59:49.777854 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777649    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 20 09:59:49.777854 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777652    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 20 09:59:49.777854 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777803    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3"} err="failed to get container status \"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3\": rpc error: code = NotFound desc = could not find container \"51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3\": container with ID starting with 51453a9c602b8d7da90fdd1081af11011ab32aaf383126d49c662449de2f26b3 not found: ID does not exist"
Jul 20 09:59:49.777854 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777826    2577 scope.go:117] "RemoveContainer" containerID="8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"
Jul 20 09:59:49.777996 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.777905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 20 09:59:49.778083 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778065    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98"} err="failed to get container status \"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98\": rpc error: code = NotFound desc = could not find container \"8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98\": container with ID starting with 8d4b6b401f70c779f90871cf4986c35614587daa8ccaac470f0a4af10038bd98 not found: ID does not exist"
Jul 20 09:59:49.778128 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778085    2577 scope.go:117] "RemoveContainer" containerID="86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"
Jul 20 09:59:49.778319 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778301    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c"} err="failed to get container status \"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c\": rpc error: code = NotFound desc = could not find container \"86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c\": container with ID starting with 86f76fd883da4fe3bc5c76209c77a213f208278c8fdc982587afb1d99db2cb4c not found: ID does not exist"
Jul 20 09:59:49.778357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778320    2577 scope.go:117] "RemoveContainer" containerID="9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"
Jul 20 09:59:49.778565 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778513    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456"} err="failed to get container status \"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456\": rpc error: code = NotFound desc = could not find container \"9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456\": container with ID starting with 9203cc5a35a070530c0a46870b8d9127829c61a3fdc2a4a456f638afa0dad456 not found: ID does not exist"
Jul 20 09:59:49.778611 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778570    2577 scope.go:117] "RemoveContainer" containerID="346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"
Jul 20 09:59:49.778816 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778798    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e"} err="failed to get container status \"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e\": rpc error: code = NotFound desc = could not find container \"346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e\": container with ID starting with 346641e3e462b6df027541a153e3f44bbd7f92ad0cba8367cdfbd85cd61f968e not found: ID does not exist"
Jul 20 09:59:49.778880 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.778816    2577 scope.go:117] "RemoveContainer" containerID="de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"
Jul 20 09:59:49.779064 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.779048    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8"} err="failed to get container status \"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8\": rpc error: code = NotFound desc = could not find container \"de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8\": container with ID starting with de10ccb0d7bc1e1f2ff376b1e211b9ae6a61e47ec36f12408746e94bf83803a8 not found: ID does not exist"
Jul 20 09:59:49.779109 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.779064    2577 scope.go:117] "RemoveContainer" containerID="4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77"
Jul 20 09:59:49.779287 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.779260    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77"} err="failed to get container status \"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77\": rpc error: code = NotFound desc = could not find container \"4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77\": container with ID starting with 4e22a6d651dfa096aca9d6cd624b3dc6a431065706f048b4f3150572a6347b77 not found: ID does not exist"
Jul 20 09:59:49.784286 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.784267    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 20 09:59:49.796893 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.796869    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7e8727-5929-4b94-bee8-25197c3e4f37" path="/var/lib/kubelet/pods/7a7e8727-5929-4b94-bee8-25197c3e4f37/volumes"
Jul 20 09:59:49.797415 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.797393    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:49.926237 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k5zd\" (UniqueName: \"kubernetes.io/projected/a0f469a0-2712-4469-9c75-560c32397361-kube-api-access-7k5zd\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926409 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926409 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f469a0-2712-4469-9c75-560c32397361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926409 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f469a0-2712-4469-9c75-560c32397361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926623 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f469a0-2712-4469-9c75-560c32397361-config-out\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926787 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926854 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926911 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.926911 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f469a0-2712-4469-9c75-560c32397361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.927024 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.926986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f469a0-2712-4469-9c75-560c32397361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.927024 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.927006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.927102 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.927076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-web-config\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:49.927170 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:49.927154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.028477 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.028477 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.028477 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f469a0-2712-4469-9c75-560c32397361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f469a0-2712-4469-9c75-560c32397361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-web-config\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k5zd\" (UniqueName: \"kubernetes.io/projected/a0f469a0-2712-4469-9c75-560c32397361-kube-api-access-7k5zd\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f469a0-2712-4469-9c75-560c32397361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f469a0-2712-4469-9c75-560c32397361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029070 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.028960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f469a0-2712-4469-9c75-560c32397361-config-out\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.029750 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.029223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f469a0-2712-4469-9c75-560c32397361-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.031756 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.031708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f469a0-2712-4469-9c75-560c32397361-config-out\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.031999 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.031974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-web-config\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.032083 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.032015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.032357 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.032289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.032491 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.032430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.032619 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.032585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f469a0-2712-4469-9c75-560c32397361-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.032774 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.032756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.032854 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.032830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f469a0-2712-4469-9c75-560c32397361-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.033295 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.033268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.033902 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.033881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f469a0-2712-4469-9c75-560c32397361-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.034164 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.034147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0f469a0-2712-4469-9c75-560c32397361-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.045802 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.045779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k5zd\" (UniqueName: \"kubernetes.io/projected/a0f469a0-2712-4469-9c75-560c32397361-kube-api-access-7k5zd\") pod \"alertmanager-main-0\" (UID: \"a0f469a0-2712-4469-9c75-560c32397361\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.084434 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.084394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:50.224385 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.224356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:50.229724 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:59:50.229698    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f469a0_2712_4469_9c75_560c32397361.slice/crio-9f1869b8a41fbb52b7472865cde049d2f5bb5295328d6da317e678ae50a6cf21 WatchSource:0}: Error finding container 9f1869b8a41fbb52b7472865cde049d2f5bb5295328d6da317e678ae50a6cf21: Status 404 returned error can't find the container with id 9f1869b8a41fbb52b7472865cde049d2f5bb5295328d6da317e678ae50a6cf21
Jul 20 09:59:50.722336 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.722302    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0f469a0-2712-4469-9c75-560c32397361" containerID="ed52bb7891ac9da43657cf8ce0a3ea6ffc618a72745acb642cf3cf34d9966acf" exitCode=0
Jul 20 09:59:50.722508 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.722397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerDied","Data":"ed52bb7891ac9da43657cf8ce0a3ea6ffc618a72745acb642cf3cf34d9966acf"}
Jul 20 09:59:50.722508 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:50.722474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerStarted","Data":"9f1869b8a41fbb52b7472865cde049d2f5bb5295328d6da317e678ae50a6cf21"}
Jul 20 09:59:51.730683 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:51.730645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerStarted","Data":"de65c5fe2261e2c8294b347a3cf7364862abad9679691a5d766209d9376a628f"}
Jul 20 09:59:51.731135 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:51.730743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerStarted","Data":"279a959cb9017eff1fa8830011eeacc36dccf5ccdddcee45de2b2f7e64db8a53"}
Jul 20 09:59:51.731135 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:51.730771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerStarted","Data":"df64c8d031b3755bf1c11f7166211bd892eb4fa6989d440f3f5a51f20d1d5ddb"}
Jul 20 09:59:51.731135 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:51.730796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerStarted","Data":"a4953de788954f841067ba66ee5fc6da25bc043c35a253b6f1c31dd80eb48d40"}
Jul 20 09:59:51.731135 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:51.730819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerStarted","Data":"266f562f5799659e15e3a81ff395e47b37a1ceaa6d1f15f173e3886405cf7d0b"}
Jul 20 09:59:51.731135 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:51.730840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0f469a0-2712-4469-9c75-560c32397361","Type":"ContainerStarted","Data":"99cda171326ee23187adf7461eb404efe1ef93e06472fb5822c2b879af96f2d8"}
Jul 20 09:59:51.764953 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:51.764885    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.746979124 podStartE2EDuration="2.764862116s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.882992ms" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:51.762344333 +0000 UTC m=+202.394610392" watchObservedRunningTime="2026-07-20 09:59:51.764862116 +0000 UTC m=+202.397128165"
Jul 20 09:59:52.135339 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.135250    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:59:52.135920 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.135886    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="prometheus" containerID="cri-o://cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c" gracePeriod=600
Jul 20 09:59:52.136408 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.136381    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10" gracePeriod=600
Jul 20 09:59:52.136665 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.136644    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy" containerID="cri-o://2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b" gracePeriod=600
Jul 20 09:59:52.136864 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.136845    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-web" containerID="cri-o://11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6" gracePeriod=600
Jul 20 09:59:52.137019 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.137003    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="thanos-sidecar" containerID="cri-o://41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb" gracePeriod=600
Jul 20 09:59:52.137171 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.137156    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="config-reloader" containerID="cri-o://d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b" gracePeriod=600
Jul 20 09:59:52.394799 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.394775    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.551442 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551408    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551442 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-metrics-client-ca\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551494    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-grpc-tls\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-web-config\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-2\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-kubelet-serving-ca-bundle\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-0\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-db\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-tls-assets\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551771    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-kube-rbac-proxy\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551828    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmjr9\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-kube-api-access-jmjr9\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551851    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-config-out\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-config\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.551921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-tls\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.552301 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551940    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-serving-certs-ca-bundle\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.552301 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.552301 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.551991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-1\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.552301 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.552030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-metrics-client-certs\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.552301 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.552063    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-trusted-ca-bundle\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.552301 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.552091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-thanos-prometheus-http-client-file\") pod \"af003670-74f2-490e-911d-dced2d7b4da1\" (UID: \"af003670-74f2-490e-911d-dced2d7b4da1\") "
Jul 20 09:59:52.552301 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.552256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.552717 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.552546    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-metrics-client-ca\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.552717 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.552574    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.554343 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.554312    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.554450 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.554346    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.554635    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-config-out" (OuterVolumeSpecName: "config-out") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.554757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.554887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.555083    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.555119    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.555125    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.555426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.555820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.555840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:59:52.556265 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.556227    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.556841 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.556461    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:52.556890 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.556860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-kube-api-access-jmjr9" (OuterVolumeSpecName: "kube-api-access-jmjr9") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "kube-api-access-jmjr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:52.557373 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.557349    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-config" (OuterVolumeSpecName: "config") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.557452 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.557400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.557712 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.557694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.565988 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.565968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-web-config" (OuterVolumeSpecName: "web-config") pod "af003670-74f2-490e-911d-dced2d7b4da1" (UID: "af003670-74f2-490e-911d-dced2d7b4da1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:52.653347 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653274    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmjr9\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-kube-api-access-jmjr9\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653347 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653301    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-config-out\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653347 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653310    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653347 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653318    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653347 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653326    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653347 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653336    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653347 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653346    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-metrics-client-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653356    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653364    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653373    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653383    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653390    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-grpc-tls\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653398    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-web-config\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653405    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653413    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653420    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af003670-74f2-490e-911d-dced2d7b4da1-prometheus-k8s-db\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653429    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af003670-74f2-490e-911d-dced2d7b4da1-tls-assets\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.653662 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.653437    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af003670-74f2-490e-911d-dced2d7b4da1-secret-kube-rbac-proxy\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 09:59:52.738798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738756    2577 generic.go:356] "Generic (PLEG): container finished" podID="af003670-74f2-490e-911d-dced2d7b4da1" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10" exitCode=0
Jul 20 09:59:52.738798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738783    2577 generic.go:356] "Generic (PLEG): container finished" podID="af003670-74f2-490e-911d-dced2d7b4da1" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b" exitCode=0
Jul 20 09:59:52.738798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738792    2577 generic.go:356] "Generic (PLEG): container finished" podID="af003670-74f2-490e-911d-dced2d7b4da1" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6" exitCode=0
Jul 20 09:59:52.738798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738800    2577 generic.go:356] "Generic (PLEG): container finished" podID="af003670-74f2-490e-911d-dced2d7b4da1" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb" exitCode=0
Jul 20 09:59:52.738798 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738806    2577 generic.go:356] "Generic (PLEG): container finished" podID="af003670-74f2-490e-911d-dced2d7b4da1" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b" exitCode=0
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738813    2577 generic.go:356] "Generic (PLEG): container finished" podID="af003670-74f2-490e-911d-dced2d7b4da1" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c" exitCode=0
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"}
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"}
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738844    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"}
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"}
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.738978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"}
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.739001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"}
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.739025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af003670-74f2-490e-911d-dced2d7b4da1","Type":"ContainerDied","Data":"40d2a9870ecfc5d087dcacd8d89036998206b314a51626b89541476fb095508f"}
Jul 20 09:59:52.739364 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.739037    2577 scope.go:117] "RemoveContainer" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.749267 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.749136    2577 scope.go:117] "RemoveContainer" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.758411 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.758391    2577 scope.go:117] "RemoveContainer" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.766613 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.766594    2577 scope.go:117] "RemoveContainer" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.772008 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.771982    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:59:52.775918 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.775903    2577 scope.go:117] "RemoveContainer" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.778981 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.778958    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:59:52.783705 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.783681    2577 scope.go:117] "RemoveContainer" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.792499 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.792480    2577 scope.go:117] "RemoveContainer" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.800172 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.800149    2577 scope.go:117] "RemoveContainer" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.800462 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:52.800444    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": container with ID starting with c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10 not found: ID does not exist" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.800510 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.800471    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"} err="failed to get container status \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": rpc error: code = NotFound desc = could not find container \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": container with ID starting with c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10 not found: ID does not exist"
Jul 20 09:59:52.800510 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.800488    2577 scope.go:117] "RemoveContainer" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.800752 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:52.800738    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": container with ID starting with 2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b not found: ID does not exist" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.800797 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.800757    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"} err="failed to get container status \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": rpc error: code = NotFound desc = could not find container \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": container with ID starting with 2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b not found: ID does not exist"
Jul 20 09:59:52.800797 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.800773    2577 scope.go:117] "RemoveContainer" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.800970 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:52.800955    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": container with ID starting with 11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6 not found: ID does not exist" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.801012 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.800972    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"} err="failed to get container status \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": rpc error: code = NotFound desc = could not find container \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": container with ID starting with 11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6 not found: ID does not exist"
Jul 20 09:59:52.801012 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.800984    2577 scope.go:117] "RemoveContainer" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.801190 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:52.801170    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": container with ID starting with 41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb not found: ID does not exist" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.801255 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801200    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"} err="failed to get container status \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": rpc error: code = NotFound desc = could not find container \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": container with ID starting with 41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb not found: ID does not exist"
Jul 20 09:59:52.801255 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801225    2577 scope.go:117] "RemoveContainer" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.801433 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:52.801418    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": container with ID starting with d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b not found: ID does not exist" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.801467 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801438    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"} err="failed to get container status \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": rpc error: code = NotFound desc = could not find container \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": container with ID starting with d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b not found: ID does not exist"
Jul 20 09:59:52.801467 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801450    2577 scope.go:117] "RemoveContainer" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.801663 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:52.801648    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": container with ID starting with cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c not found: ID does not exist" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.801717 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801667    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"} err="failed to get container status \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": rpc error: code = NotFound desc = could not find container \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": container with ID starting with cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c not found: ID does not exist"
Jul 20 09:59:52.801717 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801678    2577 scope.go:117] "RemoveContainer" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.801884 ip-10-0-137-140 kubenswrapper[2577]: E0720 09:59:52.801871    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": container with ID starting with 9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa not found: ID does not exist" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.801921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801886    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"} err="failed to get container status \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": rpc error: code = NotFound desc = could not find container \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": container with ID starting with 9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa not found: ID does not exist"
Jul 20 09:59:52.801921 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.801897    2577 scope.go:117] "RemoveContainer" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.802121 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802102    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"} err="failed to get container status \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": rpc error: code = NotFound desc = could not find container \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": container with ID starting with c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10 not found: ID does not exist"
Jul 20 09:59:52.802182 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802125    2577 scope.go:117] "RemoveContainer" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.802328 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802313    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"} err="failed to get container status \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": rpc error: code = NotFound desc = could not find container \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": container with ID starting with 2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b not found: ID does not exist"
Jul 20 09:59:52.802373 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802328    2577 scope.go:117] "RemoveContainer" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.802502 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802487    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"} err="failed to get container status \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": rpc error: code = NotFound desc = could not find container \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": container with ID starting with 11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6 not found: ID does not exist"
Jul 20 09:59:52.802584 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802501    2577 scope.go:117] "RemoveContainer" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.802709 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802694    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"} err="failed to get container status \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": rpc error: code = NotFound desc = could not find container \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": container with ID starting with 41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb not found: ID does not exist"
Jul 20 09:59:52.802752 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802708    2577 scope.go:117] "RemoveContainer" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.802881 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802864    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"} err="failed to get container status \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": rpc error: code = NotFound desc = could not find container \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": container with ID starting with d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b not found: ID does not exist"
Jul 20 09:59:52.802919 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.802883    2577 scope.go:117] "RemoveContainer" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.803086 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803068    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"} err="failed to get container status \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": rpc error: code = NotFound desc = could not find container \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": container with ID starting with cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c not found: ID does not exist"
Jul 20 09:59:52.803127 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803085    2577 scope.go:117] "RemoveContainer" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.803312 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803291    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"} err="failed to get container status \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": rpc error: code = NotFound desc = could not find container \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": container with ID starting with 9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa not found: ID does not exist"
Jul 20 09:59:52.803372 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803313    2577 scope.go:117] "RemoveContainer" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.803551 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"} err="failed to get container status \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": rpc error: code = NotFound desc = could not find container \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": container with ID starting with c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10 not found: ID does not exist"
Jul 20 09:59:52.803551 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803548    2577 scope.go:117] "RemoveContainer" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.803745 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803727    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"} err="failed to get container status \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": rpc error: code = NotFound desc = could not find container \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": container with ID starting with 2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b not found: ID does not exist"
Jul 20 09:59:52.803784 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803747    2577 scope.go:117] "RemoveContainer" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.803948 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803931    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"} err="failed to get container status \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": rpc error: code = NotFound desc = could not find container \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": container with ID starting with 11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6 not found: ID does not exist"
Jul 20 09:59:52.804005 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.803949    2577 scope.go:117] "RemoveContainer" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.804165 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804148    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"} err="failed to get container status \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": rpc error: code = NotFound desc = could not find container \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": container with ID starting with 41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb not found: ID does not exist"
Jul 20 09:59:52.804227 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804175    2577 scope.go:117] "RemoveContainer" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.804379 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804365    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"} err="failed to get container status \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": rpc error: code = NotFound desc = could not find container \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": container with ID starting with d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b not found: ID does not exist"
Jul 20 09:59:52.804424 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804379    2577 scope.go:117] "RemoveContainer" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.804603 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804585    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"} err="failed to get container status \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": rpc error: code = NotFound desc = could not find container \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": container with ID starting with cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c not found: ID does not exist"
Jul 20 09:59:52.804676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804604    2577 scope.go:117] "RemoveContainer" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.804809 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804793    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"} err="failed to get container status \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": rpc error: code = NotFound desc = could not find container \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": container with ID starting with 9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa not found: ID does not exist"
Jul 20 09:59:52.804876 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804810    2577 scope.go:117] "RemoveContainer" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.805006 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.804989    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"} err="failed to get container status \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": rpc error: code = NotFound desc = could not find container \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": container with ID starting with c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10 not found: ID does not exist"
Jul 20 09:59:52.805051 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805007    2577 scope.go:117] "RemoveContainer" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.805187 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805171    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"} err="failed to get container status \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": rpc error: code = NotFound desc = could not find container \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": container with ID starting with 2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b not found: ID does not exist"
Jul 20 09:59:52.805227 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805188    2577 scope.go:117] "RemoveContainer" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.805369 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805353    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"} err="failed to get container status \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": rpc error: code = NotFound desc = could not find container \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": container with ID starting with 11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6 not found: ID does not exist"
Jul 20 09:59:52.805436 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805371    2577 scope.go:117] "RemoveContainer" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.805588 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805569    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"} err="failed to get container status \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": rpc error: code = NotFound desc = could not find container \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": container with ID starting with 41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb not found: ID does not exist"
Jul 20 09:59:52.805639 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805590    2577 scope.go:117] "RemoveContainer" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.805769 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805753    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"} err="failed to get container status \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": rpc error: code = NotFound desc = could not find container \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": container with ID starting with d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b not found: ID does not exist"
Jul 20 09:59:52.805814 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805769    2577 scope.go:117] "RemoveContainer" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.805977 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805959    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"} err="failed to get container status \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": rpc error: code = NotFound desc = could not find container \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": container with ID starting with cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c not found: ID does not exist"
Jul 20 09:59:52.806046 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.805978    2577 scope.go:117] "RemoveContainer" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.806181 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806165    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"} err="failed to get container status \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": rpc error: code = NotFound desc = could not find container \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": container with ID starting with 9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa not found: ID does not exist"
Jul 20 09:59:52.806248 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806183    2577 scope.go:117] "RemoveContainer" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.806376 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806359    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"} err="failed to get container status \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": rpc error: code = NotFound desc = could not find container \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": container with ID starting with c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10 not found: ID does not exist"
Jul 20 09:59:52.806443 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806377    2577 scope.go:117] "RemoveContainer" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.806607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806591    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"} err="failed to get container status \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": rpc error: code = NotFound desc = could not find container \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": container with ID starting with 2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b not found: ID does not exist"
Jul 20 09:59:52.806607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806606    2577 scope.go:117] "RemoveContainer" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.806795 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806780    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"} err="failed to get container status \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": rpc error: code = NotFound desc = could not find container \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": container with ID starting with 11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6 not found: ID does not exist"
Jul 20 09:59:52.806835 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806795    2577 scope.go:117] "RemoveContainer" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.806961 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806943    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"} err="failed to get container status \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": rpc error: code = NotFound desc = could not find container \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": container with ID starting with 41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb not found: ID does not exist"
Jul 20 09:59:52.807032 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.806963    2577 scope.go:117] "RemoveContainer" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.807137 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807119    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"} err="failed to get container status \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": rpc error: code = NotFound desc = could not find container \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": container with ID starting with d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b not found: ID does not exist"
Jul 20 09:59:52.807180 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807139    2577 scope.go:117] "RemoveContainer" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.807344 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807324    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"} err="failed to get container status \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": rpc error: code = NotFound desc = could not find container \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": container with ID starting with cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c not found: ID does not exist"
Jul 20 09:59:52.807381 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807348    2577 scope.go:117] "RemoveContainer" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.807607 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807589    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"} err="failed to get container status \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": rpc error: code = NotFound desc = could not find container \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": container with ID starting with 9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa not found: ID does not exist"
Jul 20 09:59:52.807686 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807608    2577 scope.go:117] "RemoveContainer" containerID="c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"
Jul 20 09:59:52.807823 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807807    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10"} err="failed to get container status \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": rpc error: code = NotFound desc = could not find container \"c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10\": container with ID starting with c251a50b4b3ec422ed9b24b245d49b22d5d296b4bbdc85ea198064680714cd10 not found: ID does not exist"
Jul 20 09:59:52.807883 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.807824    2577 scope.go:117] "RemoveContainer" containerID="2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"
Jul 20 09:59:52.808054 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808039    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b"} err="failed to get container status \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": rpc error: code = NotFound desc = could not find container \"2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b\": container with ID starting with 2bbccb605e5058868e970937b9a3b6e698a992324de9622d24401f6df47c511b not found: ID does not exist"
Jul 20 09:59:52.808114 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808054    2577 scope.go:117] "RemoveContainer" containerID="11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"
Jul 20 09:59:52.808263 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808243    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6"} err="failed to get container status \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": rpc error: code = NotFound desc = could not find container \"11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6\": container with ID starting with 11dd9e3a504f99de9ec10f608789c68bce7b3b48fde6456053ea47f31602e9c6 not found: ID does not exist"
Jul 20 09:59:52.808318 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808264    2577 scope.go:117] "RemoveContainer" containerID="41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"
Jul 20 09:59:52.808473 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb"} err="failed to get container status \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": rpc error: code = NotFound desc = could not find container \"41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb\": container with ID starting with 41f55064e37a6f1eb10a1d49280f993a555d0436b6067e0818a143494ca697cb not found: ID does not exist"
Jul 20 09:59:52.808513 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808477    2577 scope.go:117] "RemoveContainer" containerID="d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"
Jul 20 09:59:52.808747 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808731    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b"} err="failed to get container status \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": rpc error: code = NotFound desc = could not find container \"d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b\": container with ID starting with d1b2c69d040907b8cf5c4ebbab2c4d9837aab0d7ec1671719b248f732a61797b not found: ID does not exist"
Jul 20 09:59:52.808811 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808748    2577 scope.go:117] "RemoveContainer" containerID="cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"
Jul 20 09:59:52.808996 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808976    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c"} err="failed to get container status \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": rpc error: code = NotFound desc = could not find container \"cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c\": container with ID starting with cf1001d622a3b5cb66c397bdb5089e4cc321bfea4117e5c5b4b952b85ea4700c not found: ID does not exist"
Jul 20 09:59:52.808996 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.808996    2577 scope.go:117] "RemoveContainer" containerID="9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"
Jul 20 09:59:52.809227 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.809200    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa"} err="failed to get container status \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": rpc error: code = NotFound desc = could not find container \"9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa\": container with ID starting with 9c15f2fa266cdc4f93640e8e450e4d0df9a6e3e418adc82a1e2727e5bd61e0aa not found: ID does not exist"
Jul 20 09:59:52.819875 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.819848    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:59:52.820157 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820145    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="thanos-sidecar"
Jul 20 09:59:52.820194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820159    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="thanos-sidecar"
Jul 20 09:59:52.820194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820174    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy"
Jul 20 09:59:52.820194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820180    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy"
Jul 20 09:59:52.820194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820188    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="prometheus"
Jul 20 09:59:52.820194 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820194    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="prometheus"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820202    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="init-config-reloader"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820208    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="init-config-reloader"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820215    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="config-reloader"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820220    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="config-reloader"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820226    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-web"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820231    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-web"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820240    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-thanos"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820244    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-thanos"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820289    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="prometheus"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820298    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="config-reloader"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820304    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-thanos"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820309    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820315    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="thanos-sidecar"
Jul 20 09:59:52.820333 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.820321    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af003670-74f2-490e-911d-dced2d7b4da1" containerName="kube-rbac-proxy-web"
Jul 20 09:59:52.824075 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.824058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.828550 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.828508    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 20 09:59:52.828651 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.828602    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 20 09:59:52.830014 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.829996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 20 09:59:52.830091 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.830056    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 20 09:59:52.831458 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.831444    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 20 09:59:52.831769 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.831756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 20 09:59:52.832077 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.832066    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 20 09:59:52.832758 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.832741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 20 09:59:52.835137 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.835124    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 20 09:59:52.836022 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.836009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 20 09:59:52.836382 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.836370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 20 09:59:52.836432 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.836405    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w7lvv\""
Jul 20 09:59:52.837509 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.837495    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 20 09:59:52.838046 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.838032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-89s2sppcqh6up\""
Jul 20 09:59:52.838190 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.838175    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 20 09:59:52.845473 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.845448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 20 09:59:52.862584 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.862562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:59:52.956368 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956506 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956506 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956506 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bd30472-f2aa-4c90-a501-9440e9492ab6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956506 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bd30472-f2aa-4c90-a501-9440e9492ab6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956676 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956844 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956844 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-config\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956844 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956844 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956956 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956956 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956956 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956956 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlvk\" (UniqueName: \"kubernetes.io/projected/6bd30472-f2aa-4c90-a501-9440e9492ab6-kube-api-access-xqlvk\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:52.956956 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:52.956942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057727 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057727 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bd30472-f2aa-4c90-a501-9440e9492ab6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bd30472-f2aa-4c90-a501-9440e9492ab6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.057991 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.057998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.058490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-config\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.058490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.058490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.058490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.058490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.058490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.058490 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlvk\" (UniqueName: \"kubernetes.io/projected/6bd30472-f2aa-4c90-a501-9440e9492ab6-kube-api-access-xqlvk\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.059087 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.059087 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.059087 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.058932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061103 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061246 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bd30472-f2aa-4c90-a501-9440e9492ab6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061317 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061430 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061497 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061814 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061922 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bd30472-f2aa-4c90-a501-9440e9492ab6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.061985 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.061948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.062163 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.062108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.062891 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.062868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.063050 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.063029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bd30472-f2aa-4c90-a501-9440e9492ab6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.063123 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.063081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-config\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.064003 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.063979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.064191 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.064177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.064453 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.064422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.064584 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.064501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6bd30472-f2aa-4c90-a501-9440e9492ab6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.066491 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.066474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlvk\" (UniqueName: \"kubernetes.io/projected/6bd30472-f2aa-4c90-a501-9440e9492ab6-kube-api-access-xqlvk\") pod \"prometheus-k8s-0\" (UID: \"6bd30472-f2aa-4c90-a501-9440e9492ab6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.133186 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.133154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 09:59:53.286144 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.286111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 20 09:59:53.289485 ip-10-0-137-140 kubenswrapper[2577]: W0720 09:59:53.289450    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd30472_f2aa_4c90_a501_9440e9492ab6.slice/crio-0c6600dda52bfdde54f190149fb7b649bdf0ab28e37e98f7b13f980b9cfa6367 WatchSource:0}: Error finding container 0c6600dda52bfdde54f190149fb7b649bdf0ab28e37e98f7b13f980b9cfa6367: Status 404 returned error can't find the container with id 0c6600dda52bfdde54f190149fb7b649bdf0ab28e37e98f7b13f980b9cfa6367
Jul 20 09:59:53.745246 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.745210    2577 generic.go:356] "Generic (PLEG): container finished" podID="6bd30472-f2aa-4c90-a501-9440e9492ab6" containerID="5aff3632506adee603c88c35880852d6dd20b87908008a6788d369c42559b906" exitCode=0
Jul 20 09:59:53.745698 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.745305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerDied","Data":"5aff3632506adee603c88c35880852d6dd20b87908008a6788d369c42559b906"}
Jul 20 09:59:53.745698 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.745380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerStarted","Data":"0c6600dda52bfdde54f190149fb7b649bdf0ab28e37e98f7b13f980b9cfa6367"}
Jul 20 09:59:53.797123 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:53.797095    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af003670-74f2-490e-911d-dced2d7b4da1" path="/var/lib/kubelet/pods/af003670-74f2-490e-911d-dced2d7b4da1/volumes"
Jul 20 09:59:54.754241 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:54.754207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerStarted","Data":"1cea9771a0a76d16ba75ea22939ef4ab3fa7c32796047c46b0c728f7d10b3706"}
Jul 20 09:59:54.754678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:54.754276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerStarted","Data":"a6b4a4c5484fd062aec1cf2534b116962902fe529f5267654f571ccae0b8f279"}
Jul 20 09:59:54.754678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:54.754307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerStarted","Data":"15b620ce1b31acf3d6102aa38f3635515532183803e175fe4d8b30e9a8131006"}
Jul 20 09:59:54.754678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:54.754332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerStarted","Data":"f690158d2dcb79833ea050e751d548fd36b9a2b16a1220074b8c201c29a31692"}
Jul 20 09:59:54.754678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:54.754357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerStarted","Data":"41bf9275dbade268f3563bd57341308a5d6c523bc81bc4b728774c746601653b"}
Jul 20 09:59:54.754678 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:54.754379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6bd30472-f2aa-4c90-a501-9440e9492ab6","Type":"ContainerStarted","Data":"d6a40db07dcc9990cba4647fb4edd5d3a7b1bcf571f381df729029321433c51b"}
Jul 20 09:59:54.786440 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:54.786389    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.768008356 podStartE2EDuration="2.786373368s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.365012ms" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:54.786236349 +0000 UTC m=+205.418502398" watchObservedRunningTime="2026-07-20 09:59:54.786373368 +0000 UTC m=+205.418639415"
Jul 20 09:59:58.133652 ip-10-0-137-140 kubenswrapper[2577]: I0720 09:59:58.133608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 10:00:53.134140 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:00:53.134055    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 10:00:53.150088 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:00:53.150063    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 10:00:54.005386 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:00:54.005360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 20 10:01:29.731720 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:01:29.731688    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:01:29.739658 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:01:29.739633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:01:29.741878 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:01:29.741861    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 10:04:59.609144 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.609103    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff5665d54-7bdbv"]
Jul 20 10:04:59.612624 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.612595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.618402 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.618369    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 10:04:59.619167 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.619144    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nmtb\""
Jul 20 10:04:59.619483 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.619215    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 20 10:04:59.619691 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.619637    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 10:04:59.620011 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.619262    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 10:04:59.620187 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.620164    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 10:04:59.621416 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.620075    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 20 10:04:59.621416 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.620012    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 10:04:59.626394 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.624348    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff5665d54-7bdbv"]
Jul 20 10:04:59.627147 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.627128    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 10:04:59.716865 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.716823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-service-ca\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.717038 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.716882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-oauth-config\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.717038 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.716927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-config\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.717038 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.716969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkmr\" (UniqueName: \"kubernetes.io/projected/f368923e-74a5-4de6-8cb9-ed4a1507e37c-kube-api-access-mgkmr\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.717155 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.717059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-oauth-serving-cert\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.717155 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.717094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-serving-cert\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.717155 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.717122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-trusted-ca-bundle\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.818600 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.818558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-oauth-serving-cert\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.818600 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.818600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-serving-cert\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.818858 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.818625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-trusted-ca-bundle\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.818858 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.818676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-service-ca\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.818858 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.818706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-oauth-config\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.818858 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.818719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-config\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.818858 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.818734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkmr\" (UniqueName: \"kubernetes.io/projected/f368923e-74a5-4de6-8cb9-ed4a1507e37c-kube-api-access-mgkmr\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.819415 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.819381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-oauth-serving-cert\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.819613 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.819596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-service-ca\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.819696 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.819618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-trusted-ca-bundle\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.819823 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.819802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-config\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.821217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.821196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-oauth-config\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.821314 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.821295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f368923e-74a5-4de6-8cb9-ed4a1507e37c-console-serving-cert\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.827134 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.827112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkmr\" (UniqueName: \"kubernetes.io/projected/f368923e-74a5-4de6-8cb9-ed4a1507e37c-kube-api-access-mgkmr\") pod \"console-5ff5665d54-7bdbv\" (UID: \"f368923e-74a5-4de6-8cb9-ed4a1507e37c\") " pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:04:59.928071 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:04:59.927989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:05:00.062932 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:00.062901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff5665d54-7bdbv"]
Jul 20 10:05:00.066134 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:05:00.066106    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf368923e_74a5_4de6_8cb9_ed4a1507e37c.slice/crio-654f0081bd375049092ce666dba4442074a31d9eeefcf61d117989f9274dfbd0 WatchSource:0}: Error finding container 654f0081bd375049092ce666dba4442074a31d9eeefcf61d117989f9274dfbd0: Status 404 returned error can't find the container with id 654f0081bd375049092ce666dba4442074a31d9eeefcf61d117989f9274dfbd0
Jul 20 10:05:00.980048 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:00.980012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff5665d54-7bdbv" event={"ID":"f368923e-74a5-4de6-8cb9-ed4a1507e37c","Type":"ContainerStarted","Data":"0b3f75a63217379e0b75755e27adf1f2a65d30c4b783d263df121f58bde570c5"}
Jul 20 10:05:00.980473 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:00.980058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff5665d54-7bdbv" event={"ID":"f368923e-74a5-4de6-8cb9-ed4a1507e37c","Type":"ContainerStarted","Data":"654f0081bd375049092ce666dba4442074a31d9eeefcf61d117989f9274dfbd0"}
Jul 20 10:05:01.001560 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:01.001487    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff5665d54-7bdbv" podStartSLOduration=2.001463265 podStartE2EDuration="2.001463265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:00.998007776 +0000 UTC m=+511.630273824" watchObservedRunningTime="2026-07-20 10:05:01.001463265 +0000 UTC m=+511.633729315"
Jul 20 10:05:09.928171 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:09.928136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:05:09.928754 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:09.928216    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:05:09.933089 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:09.933065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:05:10.020024 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:10.020000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ff5665d54-7bdbv"
Jul 20 10:05:37.617620 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.617580    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t8bkj"]
Jul 20 10:05:37.620956 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.620938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:37.626299 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.626277    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 20 10:05:37.626429 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.626317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 20 10:05:37.626429 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.626329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-h642t\""
Jul 20 10:05:37.653400 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.653375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t8bkj"]
Jul 20 10:05:37.757384 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.757347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390e2ab1-a378-4aec-8000-4cb84572d5e7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t8bkj\" (UID: \"390e2ab1-a378-4aec-8000-4cb84572d5e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:37.757575 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.757394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgn6g\" (UniqueName: \"kubernetes.io/projected/390e2ab1-a378-4aec-8000-4cb84572d5e7-kube-api-access-cgn6g\") pod \"cert-manager-webhook-6486ffd997-t8bkj\" (UID: \"390e2ab1-a378-4aec-8000-4cb84572d5e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:37.858120 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.858091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390e2ab1-a378-4aec-8000-4cb84572d5e7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t8bkj\" (UID: \"390e2ab1-a378-4aec-8000-4cb84572d5e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:37.858306 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.858207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgn6g\" (UniqueName: \"kubernetes.io/projected/390e2ab1-a378-4aec-8000-4cb84572d5e7-kube-api-access-cgn6g\") pod \"cert-manager-webhook-6486ffd997-t8bkj\" (UID: \"390e2ab1-a378-4aec-8000-4cb84572d5e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:37.873992 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.873933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390e2ab1-a378-4aec-8000-4cb84572d5e7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t8bkj\" (UID: \"390e2ab1-a378-4aec-8000-4cb84572d5e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:37.874936 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.874907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgn6g\" (UniqueName: \"kubernetes.io/projected/390e2ab1-a378-4aec-8000-4cb84572d5e7-kube-api-access-cgn6g\") pod \"cert-manager-webhook-6486ffd997-t8bkj\" (UID: \"390e2ab1-a378-4aec-8000-4cb84572d5e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:37.944375 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:37.944343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:38.128778 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:38.128755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t8bkj"]
Jul 20 10:05:38.131025 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:05:38.130995    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod390e2ab1_a378_4aec_8000_4cb84572d5e7.slice/crio-ba4da4af855a9b9b919ade0ac716b33c9be6230ee9bc153a9dbac0cfd0028901 WatchSource:0}: Error finding container ba4da4af855a9b9b919ade0ac716b33c9be6230ee9bc153a9dbac0cfd0028901: Status 404 returned error can't find the container with id ba4da4af855a9b9b919ade0ac716b33c9be6230ee9bc153a9dbac0cfd0028901
Jul 20 10:05:38.133024 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:38.133005    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:05:38.138590 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:38.138565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj" event={"ID":"390e2ab1-a378-4aec-8000-4cb84572d5e7","Type":"ContainerStarted","Data":"ba4da4af855a9b9b919ade0ac716b33c9be6230ee9bc153a9dbac0cfd0028901"}
Jul 20 10:05:42.160746 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:42.160711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj" event={"ID":"390e2ab1-a378-4aec-8000-4cb84572d5e7","Type":"ContainerStarted","Data":"1e73e5d114b7c0ada36fb6b843ecf069dcb258d8ee2fbfe2d449710ccc9367c1"}
Jul 20 10:05:42.161154 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:42.160932    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:42.237750 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:42.237668    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj" podStartSLOduration=2.043142629 podStartE2EDuration="5.237649761s" totalImagesPullingTime="3.194507132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:42.237596281 +0000 UTC m=+552.869862330" watchObservedRunningTime="2026-07-20 10:05:42.237649761 +0000 UTC m=+552.869915809"
Jul 20 10:05:48.166754 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:48.166724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t8bkj"
Jul 20 10:05:54.564843 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.564809    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"]
Jul 20 10:05:54.568085 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.568069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:54.573357 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.573333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 20 10:05:54.573490 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.573376    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 20 10:05:54.573844 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.573827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5hgnv\""
Jul 20 10:05:54.599025 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.598976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"]
Jul 20 10:05:54.713904 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.713871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787bw\" (UniqueName: \"kubernetes.io/projected/dbc23d97-137f-44b8-8cce-5de9186d2309-kube-api-access-787bw\") pod \"openshift-lws-operator-fd8ccff4c-7msm6\" (UID: \"dbc23d97-137f-44b8-8cce-5de9186d2309\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:54.713904 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.713904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbc23d97-137f-44b8-8cce-5de9186d2309-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7msm6\" (UID: \"dbc23d97-137f-44b8-8cce-5de9186d2309\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:54.814367 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.814332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787bw\" (UniqueName: \"kubernetes.io/projected/dbc23d97-137f-44b8-8cce-5de9186d2309-kube-api-access-787bw\") pod \"openshift-lws-operator-fd8ccff4c-7msm6\" (UID: \"dbc23d97-137f-44b8-8cce-5de9186d2309\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:54.814367 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.814370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbc23d97-137f-44b8-8cce-5de9186d2309-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7msm6\" (UID: \"dbc23d97-137f-44b8-8cce-5de9186d2309\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:54.814758 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.814739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbc23d97-137f-44b8-8cce-5de9186d2309-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7msm6\" (UID: \"dbc23d97-137f-44b8-8cce-5de9186d2309\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:54.845395 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.845326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-787bw\" (UniqueName: \"kubernetes.io/projected/dbc23d97-137f-44b8-8cce-5de9186d2309-kube-api-access-787bw\") pod \"openshift-lws-operator-fd8ccff4c-7msm6\" (UID: \"dbc23d97-137f-44b8-8cce-5de9186d2309\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:54.877126 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:54.877081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"
Jul 20 10:05:55.036373 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:55.036352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6"]
Jul 20 10:05:55.038836 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:05:55.038804    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbc23d97_137f_44b8_8cce_5de9186d2309.slice/crio-51707fa94bcf726dfdd1742869054b8bff5be8b4d5772f294cfb4899ea44adc1 WatchSource:0}: Error finding container 51707fa94bcf726dfdd1742869054b8bff5be8b4d5772f294cfb4899ea44adc1: Status 404 returned error can't find the container with id 51707fa94bcf726dfdd1742869054b8bff5be8b4d5772f294cfb4899ea44adc1
Jul 20 10:05:55.213304 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:55.213266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6" event={"ID":"dbc23d97-137f-44b8-8cce-5de9186d2309","Type":"ContainerStarted","Data":"51707fa94bcf726dfdd1742869054b8bff5be8b4d5772f294cfb4899ea44adc1"}
Jul 20 10:05:58.230345 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:58.230304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6" event={"ID":"dbc23d97-137f-44b8-8cce-5de9186d2309","Type":"ContainerStarted","Data":"7d6b6c07bb36f0dd1fd8523281342f738ba6d98ab58babb5ab5f75173a5a7a43"}
Jul 20 10:05:58.248891 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:05:58.248834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7msm6" podStartSLOduration=1.796195878 podStartE2EDuration="4.248814144s" totalImagesPullingTime="2.452618266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:05:58.247579063 +0000 UTC m=+568.879845112" watchObservedRunningTime="2026-07-20 10:05:58.248814144 +0000 UTC m=+568.881080193"
Jul 20 10:06:27.934510 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:27.934477    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"]
Jul 20 10:06:27.937602 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:27.937580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:27.944293 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:27.943965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 20 10:06:27.944293 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:27.944033    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 20 10:06:27.944293 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:27.943969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 20 10:06:27.944595 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:27.944572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j7n92\""
Jul 20 10:06:27.957519 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:27.957494    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"]
Jul 20 10:06:28.011600 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.011565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6841639-44d7-475f-a8d5-39dfdfc38b14-manager-config\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.011752 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.011604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6841639-44d7-475f-a8d5-39dfdfc38b14-metrics-cert\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.011793 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.011747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6vd\" (UniqueName: \"kubernetes.io/projected/d6841639-44d7-475f-a8d5-39dfdfc38b14-kube-api-access-dg6vd\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.011793 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.011779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6841639-44d7-475f-a8d5-39dfdfc38b14-cert\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.112737 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.112706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6vd\" (UniqueName: \"kubernetes.io/projected/d6841639-44d7-475f-a8d5-39dfdfc38b14-kube-api-access-dg6vd\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.112737 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.112742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6841639-44d7-475f-a8d5-39dfdfc38b14-cert\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.112964 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.112773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6841639-44d7-475f-a8d5-39dfdfc38b14-manager-config\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.112964 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.112818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6841639-44d7-475f-a8d5-39dfdfc38b14-metrics-cert\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.113562 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.113513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6841639-44d7-475f-a8d5-39dfdfc38b14-manager-config\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.115403 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.115381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6841639-44d7-475f-a8d5-39dfdfc38b14-cert\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.115548 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.115433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6841639-44d7-475f-a8d5-39dfdfc38b14-metrics-cert\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.120884 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.120859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6vd\" (UniqueName: \"kubernetes.io/projected/d6841639-44d7-475f-a8d5-39dfdfc38b14-kube-api-access-dg6vd\") pod \"lws-controller-manager-86dd6ccfdd-57rh6\" (UID: \"d6841639-44d7-475f-a8d5-39dfdfc38b14\") " pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.247435 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.247409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:28.380700 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:28.380663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"]
Jul 20 10:06:28.384123 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:06:28.384095    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6841639_44d7_475f_a8d5_39dfdfc38b14.slice/crio-287af345a5e95a177b2c5e224419d93f3b79232eef7674a26ee7852d82c673ee WatchSource:0}: Error finding container 287af345a5e95a177b2c5e224419d93f3b79232eef7674a26ee7852d82c673ee: Status 404 returned error can't find the container with id 287af345a5e95a177b2c5e224419d93f3b79232eef7674a26ee7852d82c673ee
Jul 20 10:06:29.370196 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:29.370150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6" event={"ID":"d6841639-44d7-475f-a8d5-39dfdfc38b14","Type":"ContainerStarted","Data":"287af345a5e95a177b2c5e224419d93f3b79232eef7674a26ee7852d82c673ee"}
Jul 20 10:06:29.872716 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:29.872692    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:06:29.873374 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:29.873351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:06:30.375949 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:30.375913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6" event={"ID":"d6841639-44d7-475f-a8d5-39dfdfc38b14","Type":"ContainerStarted","Data":"cc3908b80a4f3290e78d3216ec6902ec783e023f9b8e4ea37c3de974025b66c0"}
Jul 20 10:06:30.376418 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:30.376013    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:06:30.397412 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:30.392344    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6" podStartSLOduration=1.864423682 podStartE2EDuration="3.392323397s" totalImagesPullingTime="1.527899715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:30.390743885 +0000 UTC m=+601.023009950" watchObservedRunningTime="2026-07-20 10:06:30.392323397 +0000 UTC m=+601.024589445"
Jul 20 10:06:41.382056 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:06:41.382028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86dd6ccfdd-57rh6"
Jul 20 10:07:00.299809 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.299732    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"]
Jul 20 10:07:00.303058 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.303041    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"
Jul 20 10:07:00.307433 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.307408    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 20 10:07:00.308329 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.308305    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 20 10:07:00.313189 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.313170    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 20 10:07:00.315639 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.315620    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fwvwc\""
Jul 20 10:07:00.344201 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.344176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"]
Jul 20 10:07:00.396931 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.396894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4ln\" (UniqueName: \"kubernetes.io/projected/e6dad072-4b4d-4f90-b192-0ae8733f48be-kube-api-access-nv4ln\") pod \"dns-operator-controller-manager-65b49595d7-ld9rx\" (UID: \"e6dad072-4b4d-4f90-b192-0ae8733f48be\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"
Jul 20 10:07:00.497712 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.497668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4ln\" (UniqueName: \"kubernetes.io/projected/e6dad072-4b4d-4f90-b192-0ae8733f48be-kube-api-access-nv4ln\") pod \"dns-operator-controller-manager-65b49595d7-ld9rx\" (UID: \"e6dad072-4b4d-4f90-b192-0ae8733f48be\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"
Jul 20 10:07:00.507466 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.507435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4ln\" (UniqueName: \"kubernetes.io/projected/e6dad072-4b4d-4f90-b192-0ae8733f48be-kube-api-access-nv4ln\") pod \"dns-operator-controller-manager-65b49595d7-ld9rx\" (UID: \"e6dad072-4b4d-4f90-b192-0ae8733f48be\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"
Jul 20 10:07:00.612833 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.612744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"
Jul 20 10:07:00.759547 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:00.759489    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"]
Jul 20 10:07:00.762602 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:07:00.762572    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dad072_4b4d_4f90_b192_0ae8733f48be.slice/crio-ca6a958e31070b7b1d8a8e92ab16b5a07a10e39b25ae1672fd4aa896e91be368 WatchSource:0}: Error finding container ca6a958e31070b7b1d8a8e92ab16b5a07a10e39b25ae1672fd4aa896e91be368: Status 404 returned error can't find the container with id ca6a958e31070b7b1d8a8e92ab16b5a07a10e39b25ae1672fd4aa896e91be368
Jul 20 10:07:01.521172 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:01.521136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx" event={"ID":"e6dad072-4b4d-4f90-b192-0ae8733f48be","Type":"ContainerStarted","Data":"ca6a958e31070b7b1d8a8e92ab16b5a07a10e39b25ae1672fd4aa896e91be368"}
Jul 20 10:07:04.540407 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:04.540372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx" event={"ID":"e6dad072-4b4d-4f90-b192-0ae8733f48be","Type":"ContainerStarted","Data":"39388f576b34ce0db17016340275cd74380cad67731cf58740efed4c1033367d"}
Jul 20 10:07:04.540909 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:04.540506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"
Jul 20 10:07:04.579179 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:04.579124    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx" podStartSLOduration=1.5506112760000001 podStartE2EDuration="4.579107588s" totalImagesPullingTime="3.028496312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:04.577266697 +0000 UTC m=+635.209532747" watchObservedRunningTime="2026-07-20 10:07:04.579107588 +0000 UTC m=+635.211373636"
Jul 20 10:07:07.141084 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.141046    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"]
Jul 20 10:07:07.145019 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.144996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.151545 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.151402    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-glbwn\""
Jul 20 10:07:07.159090 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.159067    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"]
Jul 20 10:07:07.270791 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.270761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/011a6bc5-93c1-438b-a52d-ba0b9675f845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-r96fg\" (UID: \"011a6bc5-93c1-438b-a52d-ba0b9675f845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.270960 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.270808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdjf\" (UniqueName: \"kubernetes.io/projected/011a6bc5-93c1-438b-a52d-ba0b9675f845-kube-api-access-ngdjf\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-r96fg\" (UID: \"011a6bc5-93c1-438b-a52d-ba0b9675f845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.371438 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.371397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/011a6bc5-93c1-438b-a52d-ba0b9675f845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-r96fg\" (UID: \"011a6bc5-93c1-438b-a52d-ba0b9675f845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.371644 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.371459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdjf\" (UniqueName: \"kubernetes.io/projected/011a6bc5-93c1-438b-a52d-ba0b9675f845-kube-api-access-ngdjf\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-r96fg\" (UID: \"011a6bc5-93c1-438b-a52d-ba0b9675f845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.371795 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.371775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/011a6bc5-93c1-438b-a52d-ba0b9675f845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-r96fg\" (UID: \"011a6bc5-93c1-438b-a52d-ba0b9675f845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.380024 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.380006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdjf\" (UniqueName: \"kubernetes.io/projected/011a6bc5-93c1-438b-a52d-ba0b9675f845-kube-api-access-ngdjf\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-r96fg\" (UID: \"011a6bc5-93c1-438b-a52d-ba0b9675f845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.455068 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.455039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:07.617922 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:07.617894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"]
Jul 20 10:07:07.621135 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:07:07.621106    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011a6bc5_93c1_438b_a52d_ba0b9675f845.slice/crio-70cdb8bfb56192f89e3008648f8eaf1fec4c7c05e0459472f85d7ec413cfa373 WatchSource:0}: Error finding container 70cdb8bfb56192f89e3008648f8eaf1fec4c7c05e0459472f85d7ec413cfa373: Status 404 returned error can't find the container with id 70cdb8bfb56192f89e3008648f8eaf1fec4c7c05e0459472f85d7ec413cfa373
Jul 20 10:07:08.563596 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:08.563504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg" event={"ID":"011a6bc5-93c1-438b-a52d-ba0b9675f845","Type":"ContainerStarted","Data":"70cdb8bfb56192f89e3008648f8eaf1fec4c7c05e0459472f85d7ec413cfa373"}
Jul 20 10:07:12.586376 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:12.586343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg" event={"ID":"011a6bc5-93c1-438b-a52d-ba0b9675f845","Type":"ContainerStarted","Data":"abb67508757fab0cf31705b91597a04e56b534d7d22a051c6c2043860c652811"}
Jul 20 10:07:12.586811 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:12.586496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:15.547563 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:15.547513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-ld9rx"
Jul 20 10:07:15.573806 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:15.573756    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg" podStartSLOduration=4.664887484 podStartE2EDuration="8.573741839s" totalImagesPullingTime="3.908854355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:12.616763964 +0000 UTC m=+643.249030013" watchObservedRunningTime="2026-07-20 10:07:15.573741839 +0000 UTC m=+646.206007886"
Jul 20 10:07:23.594504 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:23.594472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-r96fg"
Jul 20 10:07:57.869875 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:57.869846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"]
Jul 20 10:07:57.872415 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:57.872399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:57.874786 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:57.874764    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-krh9b\""
Jul 20 10:07:57.875042 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:57.875027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 20 10:07:57.883285 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:57.883260    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"]
Jul 20 10:07:57.963025 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:57.962994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"]
Jul 20 10:07:58.031976 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.031936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2s65\" (UniqueName: \"kubernetes.io/projected/fa0823e0-b806-41b5-b9e7-62527ede7de7-kube-api-access-s2s65\") pod \"limitador-limitador-ff68ff5cb-r6rnq\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:58.032155 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.031992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa0823e0-b806-41b5-b9e7-62527ede7de7-config-file\") pod \"limitador-limitador-ff68ff5cb-r6rnq\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:58.132679 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.132593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa0823e0-b806-41b5-b9e7-62527ede7de7-config-file\") pod \"limitador-limitador-ff68ff5cb-r6rnq\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:58.132840 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.132720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2s65\" (UniqueName: \"kubernetes.io/projected/fa0823e0-b806-41b5-b9e7-62527ede7de7-kube-api-access-s2s65\") pod \"limitador-limitador-ff68ff5cb-r6rnq\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:58.133229 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.133210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa0823e0-b806-41b5-b9e7-62527ede7de7-config-file\") pod \"limitador-limitador-ff68ff5cb-r6rnq\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:58.144760 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.144737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2s65\" (UniqueName: \"kubernetes.io/projected/fa0823e0-b806-41b5-b9e7-62527ede7de7-kube-api-access-s2s65\") pod \"limitador-limitador-ff68ff5cb-r6rnq\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:58.183014 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.182981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:07:58.322999 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.322976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"]
Jul 20 10:07:58.325868 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:07:58.325826    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0823e0_b806_41b5_b9e7_62527ede7de7.slice/crio-473951d0aef0c1db415a7bb71d65a424e1740e1d7693c01878ca73bdc65ebc7b WatchSource:0}: Error finding container 473951d0aef0c1db415a7bb71d65a424e1740e1d7693c01878ca73bdc65ebc7b: Status 404 returned error can't find the container with id 473951d0aef0c1db415a7bb71d65a424e1740e1d7693c01878ca73bdc65ebc7b
Jul 20 10:07:58.811813 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:07:58.811782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq" event={"ID":"fa0823e0-b806-41b5-b9e7-62527ede7de7","Type":"ContainerStarted","Data":"473951d0aef0c1db415a7bb71d65a424e1740e1d7693c01878ca73bdc65ebc7b"}
Jul 20 10:08:02.835985 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:02.835946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq" event={"ID":"fa0823e0-b806-41b5-b9e7-62527ede7de7","Type":"ContainerStarted","Data":"7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9"}
Jul 20 10:08:02.836368 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:02.836012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:08:02.852696 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:02.852649    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq" podStartSLOduration=2.161369702 podStartE2EDuration="5.852632953s" totalImagesPullingTime="3.691263251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:02.851170174 +0000 UTC m=+693.483436222" watchObservedRunningTime="2026-07-20 10:08:02.852632953 +0000 UTC m=+693.484899001"
Jul 20 10:08:13.310620 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.310586    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"]
Jul 20 10:08:13.311143 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.310775    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq" podUID="fa0823e0-b806-41b5-b9e7-62527ede7de7" containerName="limitador" containerID="cri-o://7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9" gracePeriod=30
Jul 20 10:08:13.311545 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.311508    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:08:13.858654 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.858629    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:08:13.870340 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.870282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2s65\" (UniqueName: \"kubernetes.io/projected/fa0823e0-b806-41b5-b9e7-62527ede7de7-kube-api-access-s2s65\") pod \"fa0823e0-b806-41b5-b9e7-62527ede7de7\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") "
Jul 20 10:08:13.870450 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.870354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa0823e0-b806-41b5-b9e7-62527ede7de7-config-file\") pod \"fa0823e0-b806-41b5-b9e7-62527ede7de7\" (UID: \"fa0823e0-b806-41b5-b9e7-62527ede7de7\") "
Jul 20 10:08:13.870714 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.870691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0823e0-b806-41b5-b9e7-62527ede7de7-config-file" (OuterVolumeSpecName: "config-file") pod "fa0823e0-b806-41b5-b9e7-62527ede7de7" (UID: "fa0823e0-b806-41b5-b9e7-62527ede7de7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:08:13.872589 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.872567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0823e0-b806-41b5-b9e7-62527ede7de7-kube-api-access-s2s65" (OuterVolumeSpecName: "kube-api-access-s2s65") pod "fa0823e0-b806-41b5-b9e7-62527ede7de7" (UID: "fa0823e0-b806-41b5-b9e7-62527ede7de7"). InnerVolumeSpecName "kube-api-access-s2s65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:08:13.889514 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.889484    2577 generic.go:356] "Generic (PLEG): container finished" podID="fa0823e0-b806-41b5-b9e7-62527ede7de7" containerID="7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9" exitCode=0
Jul 20 10:08:13.889662 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.889560    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"
Jul 20 10:08:13.889662 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.889564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq" event={"ID":"fa0823e0-b806-41b5-b9e7-62527ede7de7","Type":"ContainerDied","Data":"7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9"}
Jul 20 10:08:13.889662 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.889617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq" event={"ID":"fa0823e0-b806-41b5-b9e7-62527ede7de7","Type":"ContainerDied","Data":"473951d0aef0c1db415a7bb71d65a424e1740e1d7693c01878ca73bdc65ebc7b"}
Jul 20 10:08:13.889662 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.889644    2577 scope.go:117] "RemoveContainer" containerID="7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9"
Jul 20 10:08:13.898260 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.898240    2577 scope.go:117] "RemoveContainer" containerID="7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9"
Jul 20 10:08:13.898521 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:08:13.898499    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9\": container with ID starting with 7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9 not found: ID does not exist" containerID="7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9"
Jul 20 10:08:13.898626 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.898550    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9"} err="failed to get container status \"7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9\": rpc error: code = NotFound desc = could not find container \"7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9\": container with ID starting with 7ce0be71a41e49b24708436a49c7280a21e46e0804b4bc26481a5fa715cbbaa9 not found: ID does not exist"
Jul 20 10:08:13.914713 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.914684    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"]
Jul 20 10:08:13.919964 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.919940    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-r6rnq"]
Jul 20 10:08:13.971217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.971193    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa0823e0-b806-41b5-b9e7-62527ede7de7-config-file\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:08:13.971217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:13.971214    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2s65\" (UniqueName: \"kubernetes.io/projected/fa0823e0-b806-41b5-b9e7-62527ede7de7-kube-api-access-s2s65\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:08:15.796442 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:15.796412    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0823e0-b806-41b5-b9e7-62527ede7de7" path="/var/lib/kubelet/pods/fa0823e0-b806-41b5-b9e7-62527ede7de7/volumes"
Jul 20 10:08:25.133003 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.132930    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-mjgp9"]
Jul 20 10:08:25.133432 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.133278    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0823e0-b806-41b5-b9e7-62527ede7de7" containerName="limitador"
Jul 20 10:08:25.133432 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.133289    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0823e0-b806-41b5-b9e7-62527ede7de7" containerName="limitador"
Jul 20 10:08:25.133432 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.133366    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0823e0-b806-41b5-b9e7-62527ede7de7" containerName="limitador"
Jul 20 10:08:25.143236 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.143212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.147368 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.147338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 20 10:08:25.147505 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.147400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8cj5p\""
Jul 20 10:08:25.169383 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.169356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-mjgp9"]
Jul 20 10:08:25.172814 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.172791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj86x\" (UniqueName: \"kubernetes.io/projected/ada649ba-530d-4e2d-bebf-6a247a7f10a8-kube-api-access-wj86x\") pod \"authorino-7c77957575-mjgp9\" (UID: \"ada649ba-530d-4e2d-bebf-6a247a7f10a8\") " pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.172920 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.172826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ada649ba-530d-4e2d-bebf-6a247a7f10a8-tls-cert\") pod \"authorino-7c77957575-mjgp9\" (UID: \"ada649ba-530d-4e2d-bebf-6a247a7f10a8\") " pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.274111 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.274071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj86x\" (UniqueName: \"kubernetes.io/projected/ada649ba-530d-4e2d-bebf-6a247a7f10a8-kube-api-access-wj86x\") pod \"authorino-7c77957575-mjgp9\" (UID: \"ada649ba-530d-4e2d-bebf-6a247a7f10a8\") " pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.274283 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.274118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ada649ba-530d-4e2d-bebf-6a247a7f10a8-tls-cert\") pod \"authorino-7c77957575-mjgp9\" (UID: \"ada649ba-530d-4e2d-bebf-6a247a7f10a8\") " pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.276919 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.276894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ada649ba-530d-4e2d-bebf-6a247a7f10a8-tls-cert\") pod \"authorino-7c77957575-mjgp9\" (UID: \"ada649ba-530d-4e2d-bebf-6a247a7f10a8\") " pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.284887 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.284865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj86x\" (UniqueName: \"kubernetes.io/projected/ada649ba-530d-4e2d-bebf-6a247a7f10a8-kube-api-access-wj86x\") pod \"authorino-7c77957575-mjgp9\" (UID: \"ada649ba-530d-4e2d-bebf-6a247a7f10a8\") " pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.452658 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.452633    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-mjgp9"
Jul 20 10:08:25.603470 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.603441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-mjgp9"]
Jul 20 10:08:25.607231 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:08:25.607206    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podada649ba_530d_4e2d_bebf_6a247a7f10a8.slice/crio-178fa804c09fe8b1afbe64f3b19dbd53bab47ae1031aaee568d35614f4e31d38 WatchSource:0}: Error finding container 178fa804c09fe8b1afbe64f3b19dbd53bab47ae1031aaee568d35614f4e31d38: Status 404 returned error can't find the container with id 178fa804c09fe8b1afbe64f3b19dbd53bab47ae1031aaee568d35614f4e31d38
Jul 20 10:08:25.945504 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:25.945425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-mjgp9" event={"ID":"ada649ba-530d-4e2d-bebf-6a247a7f10a8","Type":"ContainerStarted","Data":"178fa804c09fe8b1afbe64f3b19dbd53bab47ae1031aaee568d35614f4e31d38"}
Jul 20 10:08:28.970871 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:08:28.970833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-mjgp9" event={"ID":"ada649ba-530d-4e2d-bebf-6a247a7f10a8","Type":"ContainerStarted","Data":"53750bd44a09c8db8dfbe1160d37ee2d84bb6e4daa52d3f9a46655adc1ee4337"}
Jul 20 10:09:11.455640 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.455591    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-mjgp9" podStartSLOduration=44.167899019 podStartE2EDuration="46.455575347s" totalImagesPullingTime="2.287676328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:08:28.9892397 +0000 UTC m=+719.621505748" watchObservedRunningTime="2026-07-20 10:09:11.455575347 +0000 UTC m=+762.087841399"
Jul 20 10:09:11.456118 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.455886    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-rv7mc"]
Jul 20 10:09:11.461268 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.461246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.464019 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.463994    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-mxrhg\""
Jul 20 10:09:11.464133 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.464005    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 20 10:09:11.464133 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.464033    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 20 10:09:11.464133 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.464108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 20 10:09:11.471754 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.471733    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-d7pgn"]
Jul 20 10:09:11.474489 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.474469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.476931 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.476914    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 20 10:09:11.477025 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.476997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wxn9d\""
Jul 20 10:09:11.487190 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.487168    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-rv7mc"]
Jul 20 10:09:11.507079 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.507054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-d7pgn"]
Jul 20 10:09:11.581991 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.581960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpm4p\" (UniqueName: \"kubernetes.io/projected/75d105d0-7ebe-46cb-abf0-230364e31e60-kube-api-access-zpm4p\") pod \"odh-model-controller-84b54b7f97-d7pgn\" (UID: \"75d105d0-7ebe-46cb-abf0-230364e31e60\") " pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.582147 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.581998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa349bd-c1e5-4d15-a82b-db82003e356e-tls-certs\") pod \"model-serving-api-fd65d7d6-rv7mc\" (UID: \"2fa349bd-c1e5-4d15-a82b-db82003e356e\") " pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.582147 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.582021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75d105d0-7ebe-46cb-abf0-230364e31e60-cert\") pod \"odh-model-controller-84b54b7f97-d7pgn\" (UID: \"75d105d0-7ebe-46cb-abf0-230364e31e60\") " pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.582147 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.582093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4xn\" (UniqueName: \"kubernetes.io/projected/2fa349bd-c1e5-4d15-a82b-db82003e356e-kube-api-access-br4xn\") pod \"model-serving-api-fd65d7d6-rv7mc\" (UID: \"2fa349bd-c1e5-4d15-a82b-db82003e356e\") " pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.682590 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.682556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpm4p\" (UniqueName: \"kubernetes.io/projected/75d105d0-7ebe-46cb-abf0-230364e31e60-kube-api-access-zpm4p\") pod \"odh-model-controller-84b54b7f97-d7pgn\" (UID: \"75d105d0-7ebe-46cb-abf0-230364e31e60\") " pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.682746 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.682605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa349bd-c1e5-4d15-a82b-db82003e356e-tls-certs\") pod \"model-serving-api-fd65d7d6-rv7mc\" (UID: \"2fa349bd-c1e5-4d15-a82b-db82003e356e\") " pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.682746 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.682625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75d105d0-7ebe-46cb-abf0-230364e31e60-cert\") pod \"odh-model-controller-84b54b7f97-d7pgn\" (UID: \"75d105d0-7ebe-46cb-abf0-230364e31e60\") " pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.682746 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.682642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4xn\" (UniqueName: \"kubernetes.io/projected/2fa349bd-c1e5-4d15-a82b-db82003e356e-kube-api-access-br4xn\") pod \"model-serving-api-fd65d7d6-rv7mc\" (UID: \"2fa349bd-c1e5-4d15-a82b-db82003e356e\") " pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.685217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.685186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75d105d0-7ebe-46cb-abf0-230364e31e60-cert\") pod \"odh-model-controller-84b54b7f97-d7pgn\" (UID: \"75d105d0-7ebe-46cb-abf0-230364e31e60\") " pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.685217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.685189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa349bd-c1e5-4d15-a82b-db82003e356e-tls-certs\") pod \"model-serving-api-fd65d7d6-rv7mc\" (UID: \"2fa349bd-c1e5-4d15-a82b-db82003e356e\") " pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.691205 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.691181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpm4p\" (UniqueName: \"kubernetes.io/projected/75d105d0-7ebe-46cb-abf0-230364e31e60-kube-api-access-zpm4p\") pod \"odh-model-controller-84b54b7f97-d7pgn\" (UID: \"75d105d0-7ebe-46cb-abf0-230364e31e60\") " pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.695850 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.695825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4xn\" (UniqueName: \"kubernetes.io/projected/2fa349bd-c1e5-4d15-a82b-db82003e356e-kube-api-access-br4xn\") pod \"model-serving-api-fd65d7d6-rv7mc\" (UID: \"2fa349bd-c1e5-4d15-a82b-db82003e356e\") " pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.771596 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.771504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:11.784154 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.784132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:11.916039 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.915907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-rv7mc"]
Jul 20 10:09:11.919141 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:09:11.919098    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa349bd_c1e5_4d15_a82b_db82003e356e.slice/crio-b01947e74553ccbe60383e970149a1ed6aac1caeb791b57c299f2daffd9c3bf9 WatchSource:0}: Error finding container b01947e74553ccbe60383e970149a1ed6aac1caeb791b57c299f2daffd9c3bf9: Status 404 returned error can't find the container with id b01947e74553ccbe60383e970149a1ed6aac1caeb791b57c299f2daffd9c3bf9
Jul 20 10:09:11.944121 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:11.944101    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-d7pgn"]
Jul 20 10:09:11.946481 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:09:11.946457    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d105d0_7ebe_46cb_abf0_230364e31e60.slice/crio-eb99164ce7baeac1f970dcb936efa11ec7731c93aa08b2f7904c5fc8ba032871 WatchSource:0}: Error finding container eb99164ce7baeac1f970dcb936efa11ec7731c93aa08b2f7904c5fc8ba032871: Status 404 returned error can't find the container with id eb99164ce7baeac1f970dcb936efa11ec7731c93aa08b2f7904c5fc8ba032871
Jul 20 10:09:12.178947 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:12.178849    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-rv7mc" event={"ID":"2fa349bd-c1e5-4d15-a82b-db82003e356e","Type":"ContainerStarted","Data":"b01947e74553ccbe60383e970149a1ed6aac1caeb791b57c299f2daffd9c3bf9"}
Jul 20 10:09:12.180296 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:12.180256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-d7pgn" event={"ID":"75d105d0-7ebe-46cb-abf0-230364e31e60","Type":"ContainerStarted","Data":"eb99164ce7baeac1f970dcb936efa11ec7731c93aa08b2f7904c5fc8ba032871"}
Jul 20 10:09:16.206187 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:16.206148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-rv7mc" event={"ID":"2fa349bd-c1e5-4d15-a82b-db82003e356e","Type":"ContainerStarted","Data":"3ec91b271dc4d6ec6612e202b3f01f728600df06869cc72485b14036a805324a"}
Jul 20 10:09:16.206648 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:16.206303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:09:16.207631 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:16.207612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-d7pgn" event={"ID":"75d105d0-7ebe-46cb-abf0-230364e31e60","Type":"ContainerStarted","Data":"1d1233365b20416a9a2dac8a52112a7925b4b4e364776a9d7c305463d345d7bd"}
Jul 20 10:09:16.207752 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:16.207738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:16.225922 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:16.225881    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-rv7mc" podStartSLOduration=1.561839143 podStartE2EDuration="5.225867462s" totalImagesPullingTime="3.664028319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:16.223994749 +0000 UTC m=+766.856260796" watchObservedRunningTime="2026-07-20 10:09:16.225867462 +0000 UTC m=+766.858133509"
Jul 20 10:09:16.239342 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:16.239289    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-d7pgn" podStartSLOduration=1.596906718 podStartE2EDuration="5.239247067s" totalImagesPullingTime="3.642340349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:16.238956002 +0000 UTC m=+766.871222052" watchObservedRunningTime="2026-07-20 10:09:16.239247067 +0000 UTC m=+766.871513116"
Jul 20 10:09:27.214545 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:27.214511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-d7pgn"
Jul 20 10:09:27.216552 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:09:27.216519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-rv7mc"
Jul 20 10:11:11.596152 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.596118    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"]
Jul 20 10:11:11.599843 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.599822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.602467 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.602444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 20 10:11:11.602737 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.602452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 20 10:11:11.602855 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.602481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-grffw\""
Jul 20 10:11:11.602855 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.602509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 20 10:11:11.602855 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.602546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-8xm5l\""
Jul 20 10:11:11.612194 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.612172    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"]
Jul 20 10:11:11.687841 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.687804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.688026 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.687862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8blh\" (UniqueName: \"kubernetes.io/projected/d0c28c1d-c002-4e90-b3be-266e1ca1f628-kube-api-access-z8blh\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.688026 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.687892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.688026 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.687927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.688026 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.688006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789117 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789281 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789281 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789281 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8blh\" (UniqueName: \"kubernetes.io/projected/d0c28c1d-c002-4e90-b3be-266e1ca1f628-kube-api-access-z8blh\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789281 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789577 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789577 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.789728 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.789702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.791704 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.791671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.796507 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.796485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8blh\" (UniqueName: \"kubernetes.io/projected/d0c28c1d-c002-4e90-b3be-266e1ca1f628-kube-api-access-z8blh\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:11.913138 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:11.913057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:12.087391 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:12.087364    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"]
Jul 20 10:11:12.089631 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:11:12.089600    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c28c1d_c002_4e90_b3be_266e1ca1f628.slice/crio-20b46b5541bcd18236f546d1e01950ae5b3d284f60232f4cbeb7bf94aa8158de WatchSource:0}: Error finding container 20b46b5541bcd18236f546d1e01950ae5b3d284f60232f4cbeb7bf94aa8158de: Status 404 returned error can't find the container with id 20b46b5541bcd18236f546d1e01950ae5b3d284f60232f4cbeb7bf94aa8158de
Jul 20 10:11:12.091582 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:12.091560    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:11:12.786394 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:12.786350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt" event={"ID":"d0c28c1d-c002-4e90-b3be-266e1ca1f628","Type":"ContainerStarted","Data":"20b46b5541bcd18236f546d1e01950ae5b3d284f60232f4cbeb7bf94aa8158de"}
Jul 20 10:11:14.802109 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:14.802074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt" event={"ID":"d0c28c1d-c002-4e90-b3be-266e1ca1f628","Type":"ContainerStarted","Data":"08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9"}
Jul 20 10:11:14.802605 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:14.802247    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:14.818355 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:14.818291    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt" podStartSLOduration=1.283655262 podStartE2EDuration="3.818271824s" totalImagesPullingTime="2.534616562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:11:14.817753271 +0000 UTC m=+885.450019312" watchObservedRunningTime="2026-07-20 10:11:14.818271824 +0000 UTC m=+885.450537876"
Jul 20 10:11:23.180817 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.180781    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"]
Jul 20 10:11:23.184389 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.184373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.187675 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.187648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 20 10:11:23.187675 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.187657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 20 10:11:23.187846 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.187679    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jcgp7\""
Jul 20 10:11:23.187846 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.187691    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 20 10:11:23.187846 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.187757    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-j5zgq\""
Jul 20 10:11:23.195709 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.195688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"]
Jul 20 10:11:23.301790 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.301758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894g6\" (UniqueName: \"kubernetes.io/projected/9cd76e6d-13f4-4071-88be-4271d86f8051-kube-api-access-894g6\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.301945 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.301838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd76e6d-13f4-4071-88be-4271d86f8051-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.301945 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.301857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9cd76e6d-13f4-4071-88be-4271d86f8051-tokenizer-uds\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.402923 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.402889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd76e6d-13f4-4071-88be-4271d86f8051-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.402923 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.402922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9cd76e6d-13f4-4071-88be-4271d86f8051-tokenizer-uds\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.403119 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.403014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894g6\" (UniqueName: \"kubernetes.io/projected/9cd76e6d-13f4-4071-88be-4271d86f8051-kube-api-access-894g6\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.403333 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.403313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9cd76e6d-13f4-4071-88be-4271d86f8051-tokenizer-uds\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.405693 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.405664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd76e6d-13f4-4071-88be-4271d86f8051-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.410604 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.410576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894g6\" (UniqueName: \"kubernetes.io/projected/9cd76e6d-13f4-4071-88be-4271d86f8051-kube-api-access-894g6\") pod \"fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.495503 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.495474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.636300 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.636275    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"]
Jul 20 10:11:23.638377 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:11:23.638352    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd76e6d_13f4_4071_88be_4271d86f8051.slice/crio-7ea73846837ba42b424878263fddb534be351880c046a1d6e6f126d2abcfd052 WatchSource:0}: Error finding container 7ea73846837ba42b424878263fddb534be351880c046a1d6e6f126d2abcfd052: Status 404 returned error can't find the container with id 7ea73846837ba42b424878263fddb534be351880c046a1d6e6f126d2abcfd052
Jul 20 10:11:23.846568 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.846461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" event={"ID":"9cd76e6d-13f4-4071-88be-4271d86f8051","Type":"ContainerStarted","Data":"2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45"}
Jul 20 10:11:23.846568 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.846520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" event={"ID":"9cd76e6d-13f4-4071-88be-4271d86f8051","Type":"ContainerStarted","Data":"7ea73846837ba42b424878263fddb534be351880c046a1d6e6f126d2abcfd052"}
Jul 20 10:11:23.846809 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.846580    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:23.867720 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:23.867661    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" podStartSLOduration=0.867645332 podStartE2EDuration="867.645332ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:11:23.866912664 +0000 UTC m=+894.499178773" watchObservedRunningTime="2026-07-20 10:11:23.867645332 +0000 UTC m=+894.499911380"
Jul 20 10:11:29.905864 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:29.905831    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:11:29.906320 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:29.906022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:11:45.808259 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:45.808227    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:11:54.853978 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:54.853942    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:11:56.418436 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:11:56.418396    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"]
Jul 20 10:12:02.204730 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:02.204672    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"]
Jul 20 10:12:04.853998 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:04.853963    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" podUID="9cd76e6d-13f4-4071-88be-4271d86f8051" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:12:05.808209 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:05.808176    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt" podUID="d0c28c1d-c002-4e90-b3be-266e1ca1f628" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:12:11.450111 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.450065    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt" podUID="d0c28c1d-c002-4e90-b3be-266e1ca1f628" containerName="main" containerID="cri-o://08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9" gracePeriod=45
Jul 20 10:12:11.712627 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.712601    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:12:11.869357 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-cache\") pod \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") "
Jul 20 10:12:11.869357 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tls-certs\") pod \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") "
Jul 20 10:12:11.869604 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-uds\") pod \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") "
Jul 20 10:12:11.869604 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869481    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-cachi2-cache\") pod \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") "
Jul 20 10:12:11.869604 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869523    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8blh\" (UniqueName: \"kubernetes.io/projected/d0c28c1d-c002-4e90-b3be-266e1ca1f628-kube-api-access-z8blh\") pod \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\" (UID: \"d0c28c1d-c002-4e90-b3be-266e1ca1f628\") "
Jul 20 10:12:11.869735 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d0c28c1d-c002-4e90-b3be-266e1ca1f628" (UID: "d0c28c1d-c002-4e90-b3be-266e1ca1f628"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:11.869773 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d0c28c1d-c002-4e90-b3be-266e1ca1f628" (UID: "d0c28c1d-c002-4e90-b3be-266e1ca1f628"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:11.869820 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "d0c28c1d-c002-4e90-b3be-266e1ca1f628" (UID: "d0c28c1d-c002-4e90-b3be-266e1ca1f628"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:11.869875 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869860    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-uds\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:11.869918 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.869879    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tokenizer-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:11.871883 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.871859    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0c28c1d-c002-4e90-b3be-266e1ca1f628" (UID: "d0c28c1d-c002-4e90-b3be-266e1ca1f628"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:12:11.871883 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.871860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c28c1d-c002-4e90-b3be-266e1ca1f628-kube-api-access-z8blh" (OuterVolumeSpecName: "kube-api-access-z8blh") pod "d0c28c1d-c002-4e90-b3be-266e1ca1f628" (UID: "d0c28c1d-c002-4e90-b3be-266e1ca1f628"). InnerVolumeSpecName "kube-api-access-z8blh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:12:11.971294 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.971218    2577 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0c28c1d-c002-4e90-b3be-266e1ca1f628-cachi2-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:11.971294 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.971245    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8blh\" (UniqueName: \"kubernetes.io/projected/d0c28c1d-c002-4e90-b3be-266e1ca1f628-kube-api-access-z8blh\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:11.971294 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:11.971255    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0c28c1d-c002-4e90-b3be-266e1ca1f628-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:12.087248 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.087215    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0c28c1d-c002-4e90-b3be-266e1ca1f628" containerID="08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9" exitCode=0
Jul 20 10:12:12.087402 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.087257    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"
Jul 20 10:12:12.087402 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.087305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt" event={"ID":"d0c28c1d-c002-4e90-b3be-266e1ca1f628","Type":"ContainerDied","Data":"08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9"}
Jul 20 10:12:12.087402 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.087354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt" event={"ID":"d0c28c1d-c002-4e90-b3be-266e1ca1f628","Type":"ContainerDied","Data":"20b46b5541bcd18236f546d1e01950ae5b3d284f60232f4cbeb7bf94aa8158de"}
Jul 20 10:12:12.087402 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.087379    2577 scope.go:117] "RemoveContainer" containerID="08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9"
Jul 20 10:12:12.096758 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.096740    2577 scope.go:117] "RemoveContainer" containerID="08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9"
Jul 20 10:12:12.097014 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:12:12.096998    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9\": container with ID starting with 08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9 not found: ID does not exist" containerID="08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9"
Jul 20 10:12:12.097062 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.097022    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9"} err="failed to get container status \"08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9\": rpc error: code = NotFound desc = could not find container \"08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9\": container with ID starting with 08684d7edbaa1d6755d56b63c1f9dda2c2b0b05f3c6a31fe3c8b0dfab83dffc9 not found: ID does not exist"
Jul 20 10:12:12.107186 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.107162    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"]
Jul 20 10:12:12.110959 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:12.110937    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7b6tb7xt"]
Jul 20 10:12:13.796539 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:13.796506    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c28c1d-c002-4e90-b3be-266e1ca1f628" path="/var/lib/kubelet/pods/d0c28c1d-c002-4e90-b3be-266e1ca1f628/volumes"
Jul 20 10:12:14.853752 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:14.853714    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" podUID="9cd76e6d-13f4-4071-88be-4271d86f8051" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:12:17.229384 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.229347    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" podUID="9cd76e6d-13f4-4071-88be-4271d86f8051" containerName="main" containerID="cri-o://2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45" gracePeriod=45
Jul 20 10:12:17.512573 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.512548    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:12:17.519502 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.519474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd76e6d-13f4-4071-88be-4271d86f8051-tls-certs\") pod \"9cd76e6d-13f4-4071-88be-4271d86f8051\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") "
Jul 20 10:12:17.519657 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.519549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-894g6\" (UniqueName: \"kubernetes.io/projected/9cd76e6d-13f4-4071-88be-4271d86f8051-kube-api-access-894g6\") pod \"9cd76e6d-13f4-4071-88be-4271d86f8051\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") "
Jul 20 10:12:17.519657 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.519646    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9cd76e6d-13f4-4071-88be-4271d86f8051-tokenizer-uds\") pod \"9cd76e6d-13f4-4071-88be-4271d86f8051\" (UID: \"9cd76e6d-13f4-4071-88be-4271d86f8051\") "
Jul 20 10:12:17.519923 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.519893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd76e6d-13f4-4071-88be-4271d86f8051-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9cd76e6d-13f4-4071-88be-4271d86f8051" (UID: "9cd76e6d-13f4-4071-88be-4271d86f8051"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:12:17.520042 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.519967    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9cd76e6d-13f4-4071-88be-4271d86f8051-tokenizer-uds\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:17.521945 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.521911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd76e6d-13f4-4071-88be-4271d86f8051-kube-api-access-894g6" (OuterVolumeSpecName: "kube-api-access-894g6") pod "9cd76e6d-13f4-4071-88be-4271d86f8051" (UID: "9cd76e6d-13f4-4071-88be-4271d86f8051"). InnerVolumeSpecName "kube-api-access-894g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:12:17.521945 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.521920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd76e6d-13f4-4071-88be-4271d86f8051-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cd76e6d-13f4-4071-88be-4271d86f8051" (UID: "9cd76e6d-13f4-4071-88be-4271d86f8051"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:12:17.620636 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.620604    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd76e6d-13f4-4071-88be-4271d86f8051-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:17.620636 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:17.620632    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894g6\" (UniqueName: \"kubernetes.io/projected/9cd76e6d-13f4-4071-88be-4271d86f8051-kube-api-access-894g6\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:12:18.121997 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.121906    2577 generic.go:356] "Generic (PLEG): container finished" podID="9cd76e6d-13f4-4071-88be-4271d86f8051" containerID="2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45" exitCode=0
Jul 20 10:12:18.121997 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.121946    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"
Jul 20 10:12:18.122234 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.121996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" event={"ID":"9cd76e6d-13f4-4071-88be-4271d86f8051","Type":"ContainerDied","Data":"2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45"}
Jul 20 10:12:18.122234 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.122052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w" event={"ID":"9cd76e6d-13f4-4071-88be-4271d86f8051","Type":"ContainerDied","Data":"7ea73846837ba42b424878263fddb534be351880c046a1d6e6f126d2abcfd052"}
Jul 20 10:12:18.122234 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.122080    2577 scope.go:117] "RemoveContainer" containerID="2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45"
Jul 20 10:12:18.130846 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.130827    2577 scope.go:117] "RemoveContainer" containerID="2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45"
Jul 20 10:12:18.131122 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:12:18.131105    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45\": container with ID starting with 2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45 not found: ID does not exist" containerID="2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45"
Jul 20 10:12:18.131172 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.131131    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45"} err="failed to get container status \"2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45\": rpc error: code = NotFound desc = could not find container \"2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45\": container with ID starting with 2c26bc3193aa90b5ec76abdb018548d65f7f9742bf29edd9abdacb5ed1a7be45 not found: ID does not exist"
Jul 20 10:12:18.141285 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.141263    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"]
Jul 20 10:12:18.144549 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:18.144508    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-66d7c96dc9-d2s5w"]
Jul 20 10:12:19.797056 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:19.797022    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd76e6d-13f4-4071-88be-4271d86f8051" path="/var/lib/kubelet/pods/9cd76e6d-13f4-4071-88be-4271d86f8051/volumes"
Jul 20 10:12:23.226457 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.226424    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"]
Jul 20 10:12:23.226888 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.226869    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c28c1d-c002-4e90-b3be-266e1ca1f628" containerName="main"
Jul 20 10:12:23.226888 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.226883    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c28c1d-c002-4e90-b3be-266e1ca1f628" containerName="main"
Jul 20 10:12:23.226957 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.226919    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd76e6d-13f4-4071-88be-4271d86f8051" containerName="main"
Jul 20 10:12:23.226957 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.226924    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd76e6d-13f4-4071-88be-4271d86f8051" containerName="main"
Jul 20 10:12:23.227022 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.226994    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c28c1d-c002-4e90-b3be-266e1ca1f628" containerName="main"
Jul 20 10:12:23.227022 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.227005    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd76e6d-13f4-4071-88be-4271d86f8051" containerName="main"
Jul 20 10:12:23.231482 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.231455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.234709 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.234691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-mhjd2\""
Jul 20 10:12:23.234831 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.234796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 20 10:12:23.234831 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.234809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 20 10:12:23.234947 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.234796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-rbffz\""
Jul 20 10:12:23.234947 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.234914    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 20 10:12:23.255187 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.255157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"]
Jul 20 10:12:23.268591 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.268566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.268731 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.268610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f821d04f-f495-4e18-9e94-7a6d1599d581-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.268731 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.268642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.268839 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.268726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.268839 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.268762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.268839 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.268793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97j8\" (UniqueName: \"kubernetes.io/projected/f821d04f-f495-4e18-9e94-7a6d1599d581-kube-api-access-p97j8\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.369611 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.369578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97j8\" (UniqueName: \"kubernetes.io/projected/f821d04f-f495-4e18-9e94-7a6d1599d581-kube-api-access-p97j8\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.369798 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.369673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.369798 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.369697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f821d04f-f495-4e18-9e94-7a6d1599d581-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.369798 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.369716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.369798 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.369744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.369798 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.369759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.370188 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.370163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.370263 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.370190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.370263 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.370241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.372168 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.372144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.372294 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.372279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f821d04f-f495-4e18-9e94-7a6d1599d581-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.378556 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.378509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97j8\" (UniqueName: \"kubernetes.io/projected/f821d04f-f495-4e18-9e94-7a6d1599d581-kube-api-access-p97j8\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.540402 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.540321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:23.686113 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:23.686091    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"]
Jul 20 10:12:23.688848 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:12:23.688817    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf821d04f_f495_4e18_9e94_7a6d1599d581.slice/crio-a7507f5ef43ff2e2886d1ddde695ac3d8797fd96ab416581f827d0b523f9e311 WatchSource:0}: Error finding container a7507f5ef43ff2e2886d1ddde695ac3d8797fd96ab416581f827d0b523f9e311: Status 404 returned error can't find the container with id a7507f5ef43ff2e2886d1ddde695ac3d8797fd96ab416581f827d0b523f9e311
Jul 20 10:12:24.153951 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:24.153914    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" event={"ID":"f821d04f-f495-4e18-9e94-7a6d1599d581","Type":"ContainerStarted","Data":"a7507f5ef43ff2e2886d1ddde695ac3d8797fd96ab416581f827d0b523f9e311"}
Jul 20 10:12:25.160314 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:25.160269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" event={"ID":"f821d04f-f495-4e18-9e94-7a6d1599d581","Type":"ContainerStarted","Data":"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"}
Jul 20 10:12:25.160742 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:25.160473    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:27.173596 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:27.173482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" event={"ID":"f821d04f-f495-4e18-9e94-7a6d1599d581","Type":"ContainerStarted","Data":"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"}
Jul 20 10:12:27.193858 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:27.193813    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" podStartSLOduration=1.248268541 podStartE2EDuration="4.193796441s" totalImagesPullingTime="2.9455279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:12:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:12:27.191607203 +0000 UTC m=+957.823873250" watchObservedRunningTime="2026-07-20 10:12:27.193796441 +0000 UTC m=+957.826062488"
Jul 20 10:12:27.541325 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:27.541287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:28.946789 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.946759    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"]
Jul 20 10:12:28.950770 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.950748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:28.952926 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.952901    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 20 10:12:28.953055 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.952901    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-5wkw5\""
Jul 20 10:12:28.953499 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.953479    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 20 10:12:28.953640 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.953521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 20 10:12:28.953712 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.953659    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-lzhg5\""
Jul 20 10:12:28.961332 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:28.961305    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"]
Jul 20 10:12:29.025914 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.025884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlg58\" (UniqueName: \"kubernetes.io/projected/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-kube-api-access-qlg58\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.026069 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.025941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.026069 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.025968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tokenizer-uds\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.127446 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.127409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlg58\" (UniqueName: \"kubernetes.io/projected/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-kube-api-access-qlg58\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.127637 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.127467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.127637 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.127490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tokenizer-uds\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.127869 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.127852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tokenizer-uds\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.130147 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.130118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.134926 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.134902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlg58\" (UniqueName: \"kubernetes.io/projected/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-kube-api-access-qlg58\") pod \"fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.266088 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.266050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:29.411842 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:29.411775    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"]
Jul 20 10:12:29.414137 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:12:29.414112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7a13fd_8810_42d7_aea2_a4f12bd8f15c.slice/crio-29a1c8f3d3c2f55c9203bd2d1073c55052c9673dfc284b560a8bdc2ad6daf0c3 WatchSource:0}: Error finding container 29a1c8f3d3c2f55c9203bd2d1073c55052c9673dfc284b560a8bdc2ad6daf0c3: Status 404 returned error can't find the container with id 29a1c8f3d3c2f55c9203bd2d1073c55052c9673dfc284b560a8bdc2ad6daf0c3
Jul 20 10:12:30.197762 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:30.197726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" event={"ID":"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c","Type":"ContainerStarted","Data":"ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34"}
Jul 20 10:12:30.198141 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:30.197802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" event={"ID":"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c","Type":"ContainerStarted","Data":"29a1c8f3d3c2f55c9203bd2d1073c55052c9673dfc284b560a8bdc2ad6daf0c3"}
Jul 20 10:12:30.198141 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:30.197854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:12:30.262439 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:30.262326    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" podStartSLOduration=2.262310726 podStartE2EDuration="2.262310726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:12:30.260652578 +0000 UTC m=+960.892918627" watchObservedRunningTime="2026-07-20 10:12:30.262310726 +0000 UTC m=+960.894576774"
Jul 20 10:12:33.541449 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:33.541415    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:33.553900 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:33.553878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:34.229211 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:34.229183    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:12:35.234665 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:12:35.234639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:13:01.204603 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:01.204478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:13:07.937126 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:07.937093    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"]
Jul 20 10:13:11.204939 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:11.204902    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" podUID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:13:21.204879 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:21.204845    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" podUID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:13:22.963609 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:22.963570    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" podUID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" containerName="main" containerID="cri-o://ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34" gracePeriod=45
Jul 20 10:13:23.208453 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.208429    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:13:23.341246 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.341161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tls-certs\") pod \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") "
Jul 20 10:13:23.341395 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.341258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlg58\" (UniqueName: \"kubernetes.io/projected/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-kube-api-access-qlg58\") pod \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") "
Jul 20 10:13:23.341395 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.341306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tokenizer-uds\") pod \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\" (UID: \"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c\") "
Jul 20 10:13:23.341638 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.341613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" (UID: "2b7a13fd-8810-42d7-aea2-a4f12bd8f15c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:13:23.343629 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.343602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" (UID: "2b7a13fd-8810-42d7-aea2-a4f12bd8f15c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:13:23.343737 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.343679    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-kube-api-access-qlg58" (OuterVolumeSpecName: "kube-api-access-qlg58") pod "2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" (UID: "2b7a13fd-8810-42d7-aea2-a4f12bd8f15c"). InnerVolumeSpecName "kube-api-access-qlg58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:13:23.442960 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.442927    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:13:23.442960 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.442954    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlg58\" (UniqueName: \"kubernetes.io/projected/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-kube-api-access-qlg58\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:13:23.442960 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.442964    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c-tokenizer-uds\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:13:23.466397 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.466362    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" containerID="ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34" exitCode=0
Jul 20 10:13:23.466573 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.466414    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"
Jul 20 10:13:23.466573 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.466447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" event={"ID":"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c","Type":"ContainerDied","Data":"ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34"}
Jul 20 10:13:23.466573 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.466503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d" event={"ID":"2b7a13fd-8810-42d7-aea2-a4f12bd8f15c","Type":"ContainerDied","Data":"29a1c8f3d3c2f55c9203bd2d1073c55052c9673dfc284b560a8bdc2ad6daf0c3"}
Jul 20 10:13:23.466573 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.466552    2577 scope.go:117] "RemoveContainer" containerID="ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34"
Jul 20 10:13:23.475928 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.475913    2577 scope.go:117] "RemoveContainer" containerID="ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34"
Jul 20 10:13:23.476234 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:13:23.476216    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34\": container with ID starting with ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34 not found: ID does not exist" containerID="ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34"
Jul 20 10:13:23.476302 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.476244    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34"} err="failed to get container status \"ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34\": rpc error: code = NotFound desc = could not find container \"ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34\": container with ID starting with ba7dbfbcd7edb4f375d86bc7fa51bcdcc6c357092116852d8c786b8a2030dd34 not found: ID does not exist"
Jul 20 10:13:23.490633 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.490612    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"]
Jul 20 10:13:23.493838 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.493818    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7fdbc65cd4-5t56d"]
Jul 20 10:13:23.802449 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:13:23.802410    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" path="/var/lib/kubelet/pods/2b7a13fd-8810-42d7-aea2-a4f12bd8f15c/volumes"
Jul 20 10:14:06.627666 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:06.627630    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"]
Jul 20 10:14:06.637227 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:14:06.637189    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:14:06.637227 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:06.637227 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:06.637227 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698" cmd=["/bin/sleep","15"]
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:06.637248    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" message="" err=<
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:14:06.637260    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="main" containerID="cri-o://3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"
Jul 20 10:14:06.637488 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:06.637307    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="main" containerID="cri-o://3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698" gracePeriod=60
Jul 20 10:14:06.749849 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:06.749809    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="llm-d-routing-sidecar" containerID="cri-o://ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5" gracePeriod=60
Jul 20 10:14:07.001895 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.001872    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:14:07.152863 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.152836    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-dshm\") pod \"f821d04f-f495-4e18-9e94-7a6d1599d581\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") "
Jul 20 10:14:07.153031 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.152894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f821d04f-f495-4e18-9e94-7a6d1599d581-tls-certs\") pod \"f821d04f-f495-4e18-9e94-7a6d1599d581\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") "
Jul 20 10:14:07.153031 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.152911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-home\") pod \"f821d04f-f495-4e18-9e94-7a6d1599d581\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") "
Jul 20 10:14:07.153031 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.152928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-model-cache\") pod \"f821d04f-f495-4e18-9e94-7a6d1599d581\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") "
Jul 20 10:14:07.153031 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.152956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p97j8\" (UniqueName: \"kubernetes.io/projected/f821d04f-f495-4e18-9e94-7a6d1599d581-kube-api-access-p97j8\") pod \"f821d04f-f495-4e18-9e94-7a6d1599d581\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") "
Jul 20 10:14:07.153031 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.153033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-tmp-dir\") pod \"f821d04f-f495-4e18-9e94-7a6d1599d581\" (UID: \"f821d04f-f495-4e18-9e94-7a6d1599d581\") "
Jul 20 10:14:07.153301 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.153245    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-model-cache" (OuterVolumeSpecName: "model-cache") pod "f821d04f-f495-4e18-9e94-7a6d1599d581" (UID: "f821d04f-f495-4e18-9e94-7a6d1599d581"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.153301 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.153261    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-home" (OuterVolumeSpecName: "home") pod "f821d04f-f495-4e18-9e94-7a6d1599d581" (UID: "f821d04f-f495-4e18-9e94-7a6d1599d581"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.153420 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.153393    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f821d04f-f495-4e18-9e94-7a6d1599d581" (UID: "f821d04f-f495-4e18-9e94-7a6d1599d581"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.155445 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.155417    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-dshm" (OuterVolumeSpecName: "dshm") pod "f821d04f-f495-4e18-9e94-7a6d1599d581" (UID: "f821d04f-f495-4e18-9e94-7a6d1599d581"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:14:07.155576 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.155478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f821d04f-f495-4e18-9e94-7a6d1599d581-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f821d04f-f495-4e18-9e94-7a6d1599d581" (UID: "f821d04f-f495-4e18-9e94-7a6d1599d581"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:14:07.155576 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.155542    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f821d04f-f495-4e18-9e94-7a6d1599d581-kube-api-access-p97j8" (OuterVolumeSpecName: "kube-api-access-p97j8") pod "f821d04f-f495-4e18-9e94-7a6d1599d581" (UID: "f821d04f-f495-4e18-9e94-7a6d1599d581"). InnerVolumeSpecName "kube-api-access-p97j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:14:07.254700 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.254655    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f821d04f-f495-4e18-9e94-7a6d1599d581-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.254700 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.254694    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.254700 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.254704    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.254700 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.254712    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97j8\" (UniqueName: \"kubernetes.io/projected/f821d04f-f495-4e18-9e94-7a6d1599d581-kube-api-access-p97j8\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.255023 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.254720    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.255023 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.254731    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f821d04f-f495-4e18-9e94-7a6d1599d581-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:14:07.684452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.684358    2577 generic.go:356] "Generic (PLEG): container finished" podID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerID="3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698" exitCode=0
Jul 20 10:14:07.684452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.684388    2577 generic.go:356] "Generic (PLEG): container finished" podID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerID="ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5" exitCode=0
Jul 20 10:14:07.684452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.684414    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"
Jul 20 10:14:07.684452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.684434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" event={"ID":"f821d04f-f495-4e18-9e94-7a6d1599d581","Type":"ContainerDied","Data":"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"}
Jul 20 10:14:07.685110 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.684503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" event={"ID":"f821d04f-f495-4e18-9e94-7a6d1599d581","Type":"ContainerDied","Data":"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"}
Jul 20 10:14:07.685110 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.684522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj" event={"ID":"f821d04f-f495-4e18-9e94-7a6d1599d581","Type":"ContainerDied","Data":"a7507f5ef43ff2e2886d1ddde695ac3d8797fd96ab416581f827d0b523f9e311"}
Jul 20 10:14:07.685110 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.684556    2577 scope.go:117] "RemoveContainer" containerID="3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"
Jul 20 10:14:07.693967 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.693950    2577 scope.go:117] "RemoveContainer" containerID="ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"
Jul 20 10:14:07.701629 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.701612    2577 scope.go:117] "RemoveContainer" containerID="3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"
Jul 20 10:14:07.701869 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:14:07.701851    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698\": container with ID starting with 3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698 not found: ID does not exist" containerID="3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"
Jul 20 10:14:07.701931 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.701879    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"} err="failed to get container status \"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698\": rpc error: code = NotFound desc = could not find container \"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698\": container with ID starting with 3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698 not found: ID does not exist"
Jul 20 10:14:07.701931 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.701896    2577 scope.go:117] "RemoveContainer" containerID="ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"
Jul 20 10:14:07.702131 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:14:07.702117    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5\": container with ID starting with ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5 not found: ID does not exist" containerID="ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"
Jul 20 10:14:07.702169 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.702137    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"} err="failed to get container status \"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5\": rpc error: code = NotFound desc = could not find container \"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5\": container with ID starting with ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5 not found: ID does not exist"
Jul 20 10:14:07.702169 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.702149    2577 scope.go:117] "RemoveContainer" containerID="3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"
Jul 20 10:14:07.702361 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.702344    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698"} err="failed to get container status \"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698\": rpc error: code = NotFound desc = could not find container \"3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698\": container with ID starting with 3f8090f9376c8a34b22ef5a96b7ad69709d0bbd9af6443064e2ab6870ed80698 not found: ID does not exist"
Jul 20 10:14:07.702407 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.702362    2577 scope.go:117] "RemoveContainer" containerID="ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"
Jul 20 10:14:07.702594 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.702574    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5"} err="failed to get container status \"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5\": rpc error: code = NotFound desc = could not find container \"ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5\": container with ID starting with ff656c8a14b1224064d5701aa9d6eae96b97558e057a6bacd1786b699915c1a5 not found: ID does not exist"
Jul 20 10:14:07.717739 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.717715    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"]
Jul 20 10:14:07.719290 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.719267    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-jt7cj"]
Jul 20 10:14:07.796698 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:07.796659    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" path="/var/lib/kubelet/pods/f821d04f-f495-4e18-9e94-7a6d1599d581/volumes"
Jul 20 10:14:30.637790 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.637756    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"]
Jul 20 10:14:30.638196 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638134    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="main"
Jul 20 10:14:30.638196 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638144    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="main"
Jul 20 10:14:30.638196 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="llm-d-routing-sidecar"
Jul 20 10:14:30.638196 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638165    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="llm-d-routing-sidecar"
Jul 20 10:14:30.638196 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638171    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" containerName="main"
Jul 20 10:14:30.638196 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638177    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" containerName="main"
Jul 20 10:14:30.638408 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638237    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="main"
Jul 20 10:14:30.638408 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638248    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f821d04f-f495-4e18-9e94-7a6d1599d581" containerName="llm-d-routing-sidecar"
Jul 20 10:14:30.638408 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.638254    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7a13fd-8810-42d7-aea2-a4f12bd8f15c" containerName="main"
Jul 20 10:14:30.642376 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.642356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.645200 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.645177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-l5dxl\""
Jul 20 10:14:30.645988 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.645968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 20 10:14:30.646212 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.646196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 20 10:14:30.646881 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.646867    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 20 10:14:30.646970 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.646943    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-fxpnp\""
Jul 20 10:14:30.663103 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.663076    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"]
Jul 20 10:14:30.666276 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.666250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.666416 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.666298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.666416 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.666343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8kp\" (UniqueName: \"kubernetes.io/projected/7844dde4-c9d2-4497-98b1-747dfea4b3bc-kube-api-access-9h8kp\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.666416 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.666368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.666584 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.666504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.666584 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.666523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767000 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.766961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767000 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.766999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8kp\" (UniqueName: \"kubernetes.io/projected/7844dde4-c9d2-4497-98b1-747dfea4b3bc-kube-api-access-9h8kp\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767268 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.767015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767268 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.767097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767268 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.767121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767268 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.767173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767580 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.767559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767644 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.767583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.767690 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.767656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.769456 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.769426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.769591 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.769459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.775292 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.775267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8kp\" (UniqueName: \"kubernetes.io/projected/7844dde4-c9d2-4497-98b1-747dfea4b3bc-kube-api-access-9h8kp\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:30.952109 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:30.952025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:31.095927 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:31.095884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"]
Jul 20 10:14:31.099565 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:14:31.099509    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7844dde4_c9d2_4497_98b1_747dfea4b3bc.slice/crio-99b3d437e56cb3bd6fb564451b1132e7da5da649cfb9c7f5cf81ca5b30edd4f0 WatchSource:0}: Error finding container 99b3d437e56cb3bd6fb564451b1132e7da5da649cfb9c7f5cf81ca5b30edd4f0: Status 404 returned error can't find the container with id 99b3d437e56cb3bd6fb564451b1132e7da5da649cfb9c7f5cf81ca5b30edd4f0
Jul 20 10:14:31.813918 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:31.813882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" event={"ID":"7844dde4-c9d2-4497-98b1-747dfea4b3bc","Type":"ContainerStarted","Data":"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"}
Jul 20 10:14:31.814335 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:31.813947    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:31.814335 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:31.813973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" event={"ID":"7844dde4-c9d2-4497-98b1-747dfea4b3bc","Type":"ContainerStarted","Data":"99b3d437e56cb3bd6fb564451b1132e7da5da649cfb9c7f5cf81ca5b30edd4f0"}
Jul 20 10:14:32.820274 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:32.820236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" event={"ID":"7844dde4-c9d2-4497-98b1-747dfea4b3bc","Type":"ContainerStarted","Data":"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"}
Jul 20 10:14:32.841635 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:32.841583    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" podStartSLOduration=2.841566551 podStartE2EDuration="2.841566551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:14:32.839199504 +0000 UTC m=+1083.471465551" watchObservedRunningTime="2026-07-20 10:14:32.841566551 +0000 UTC m=+1083.473832598"
Jul 20 10:14:32.953081 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:32.953042    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:40.952626 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:40.952587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:40.965010 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:40.964983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:41.875489 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:41.875461    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:14:41.876893 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:14:41.876875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:15:11.414992 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:11.414952    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"]
Jul 20 10:15:11.425324 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:15:11.425275    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:15:11.425324 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:15:11.425324 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:15:11.425324 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad" cmd=["/bin/sleep","15"]
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:11.425338    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" message="" err=<
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:15:11.425351    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="main" containerID="cri-o://8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"
Jul 20 10:15:11.425681 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:11.425406    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="main" containerID="cri-o://8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad" gracePeriod=60
Jul 20 10:15:11.642554 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:11.642479    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="llm-d-routing-sidecar" containerID="cri-o://015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d" gracePeriod=60
Jul 20 10:15:11.895119 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:11.895089    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:15:12.020394 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.020293    2577 generic.go:356] "Generic (PLEG): container finished" podID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerID="8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad" exitCode=0
Jul 20 10:15:12.020394 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.020329    2577 generic.go:356] "Generic (PLEG): container finished" podID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerID="015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d" exitCode=0
Jul 20 10:15:12.020394 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.020343    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"
Jul 20 10:15:12.020394 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.020378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" event={"ID":"7844dde4-c9d2-4497-98b1-747dfea4b3bc","Type":"ContainerDied","Data":"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"}
Jul 20 10:15:12.020669 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.020433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" event={"ID":"7844dde4-c9d2-4497-98b1-747dfea4b3bc","Type":"ContainerDied","Data":"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"}
Jul 20 10:15:12.020669 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.020450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" event={"ID":"7844dde4-c9d2-4497-98b1-747dfea4b3bc","Type":"ContainerDied","Data":"99b3d437e56cb3bd6fb564451b1132e7da5da649cfb9c7f5cf81ca5b30edd4f0"}
Jul 20 10:15:12.020669 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.020482    2577 scope.go:117] "RemoveContainer" containerID="8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"
Jul 20 10:15:12.025765 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.025743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-dshm\") pod \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") "
Jul 20 10:15:12.025880 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.025781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-home\") pod \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") "
Jul 20 10:15:12.025880 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.025815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h8kp\" (UniqueName: \"kubernetes.io/projected/7844dde4-c9d2-4497-98b1-747dfea4b3bc-kube-api-access-9h8kp\") pod \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") "
Jul 20 10:15:12.025968 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.025894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tls-certs\") pod \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") "
Jul 20 10:15:12.025968 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.025929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-model-cache\") pod \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") "
Jul 20 10:15:12.026078 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.026007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tmp-dir\") pod \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\" (UID: \"7844dde4-c9d2-4497-98b1-747dfea4b3bc\") "
Jul 20 10:15:12.026078 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.026034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-home" (OuterVolumeSpecName: "home") pod "7844dde4-c9d2-4497-98b1-747dfea4b3bc" (UID: "7844dde4-c9d2-4497-98b1-747dfea4b3bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:12.026443 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.026231    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "7844dde4-c9d2-4497-98b1-747dfea4b3bc" (UID: "7844dde4-c9d2-4497-98b1-747dfea4b3bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:12.026443 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.026330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7844dde4-c9d2-4497-98b1-747dfea4b3bc" (UID: "7844dde4-c9d2-4497-98b1-747dfea4b3bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:12.026443 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.026401    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:15:12.026443 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.026421    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:15:12.028613 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.028586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-dshm" (OuterVolumeSpecName: "dshm") pod "7844dde4-c9d2-4497-98b1-747dfea4b3bc" (UID: "7844dde4-c9d2-4497-98b1-747dfea4b3bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:15:12.028739 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.028641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7844dde4-c9d2-4497-98b1-747dfea4b3bc" (UID: "7844dde4-c9d2-4497-98b1-747dfea4b3bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:15:12.028739 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.028660    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7844dde4-c9d2-4497-98b1-747dfea4b3bc-kube-api-access-9h8kp" (OuterVolumeSpecName: "kube-api-access-9h8kp") pod "7844dde4-c9d2-4497-98b1-747dfea4b3bc" (UID: "7844dde4-c9d2-4497-98b1-747dfea4b3bc"). InnerVolumeSpecName "kube-api-access-9h8kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:15:12.030853 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.030836    2577 scope.go:117] "RemoveContainer" containerID="015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"
Jul 20 10:15:12.040842 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.040825    2577 scope.go:117] "RemoveContainer" containerID="8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"
Jul 20 10:15:12.041156 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:15:12.041132    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad\": container with ID starting with 8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad not found: ID does not exist" containerID="8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"
Jul 20 10:15:12.041245 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.041163    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"} err="failed to get container status \"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad\": rpc error: code = NotFound desc = could not find container \"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad\": container with ID starting with 8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad not found: ID does not exist"
Jul 20 10:15:12.041245 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.041181    2577 scope.go:117] "RemoveContainer" containerID="015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"
Jul 20 10:15:12.041423 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:15:12.041402    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d\": container with ID starting with 015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d not found: ID does not exist" containerID="015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"
Jul 20 10:15:12.041493 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.041437    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"} err="failed to get container status \"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d\": rpc error: code = NotFound desc = could not find container \"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d\": container with ID starting with 015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d not found: ID does not exist"
Jul 20 10:15:12.041493 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.041462    2577 scope.go:117] "RemoveContainer" containerID="8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"
Jul 20 10:15:12.041795 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.041774    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad"} err="failed to get container status \"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad\": rpc error: code = NotFound desc = could not find container \"8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad\": container with ID starting with 8c1eab1496198bef87587fa724c61e3f625992c2ef43cd76ff12af1be9b246ad not found: ID does not exist"
Jul 20 10:15:12.041855 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.041797    2577 scope.go:117] "RemoveContainer" containerID="015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"
Jul 20 10:15:12.042029 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.042009    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d"} err="failed to get container status \"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d\": rpc error: code = NotFound desc = could not find container \"015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d\": container with ID starting with 015dc7efcf2950fabbf80c5e2268cd2650d516d92b5a361965021253f801d41d not found: ID does not exist"
Jul 20 10:15:12.126936 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.126903    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h8kp\" (UniqueName: \"kubernetes.io/projected/7844dde4-c9d2-4497-98b1-747dfea4b3bc-kube-api-access-9h8kp\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:15:12.126936 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.126931    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:15:12.127142 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.126946    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:15:12.127142 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.126957    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7844dde4-c9d2-4497-98b1-747dfea4b3bc-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:15:12.344880 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.344844    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"]
Jul 20 10:15:12.350684 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.350658    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp"]
Jul 20 10:15:12.864990 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:12.864947    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:15:13.796493 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:13.796449    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" path="/var/lib/kubelet/pods/7844dde4-c9d2-4497-98b1-747dfea4b3bc/volumes"
Jul 20 10:15:16.865424 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:16.865376    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-jqcpp" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:15:38.632134 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.632100    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"]
Jul 20 10:15:38.632547 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.632518    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="main"
Jul 20 10:15:38.632607 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.632549    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="main"
Jul 20 10:15:38.632607 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.632560    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="llm-d-routing-sidecar"
Jul 20 10:15:38.632607 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.632566    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="llm-d-routing-sidecar"
Jul 20 10:15:38.632938 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.632635    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="main"
Jul 20 10:15:38.632938 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.632649    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7844dde4-c9d2-4497-98b1-747dfea4b3bc" containerName="llm-d-routing-sidecar"
Jul 20 10:15:38.636856 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.636838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.652626 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.652598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 20 10:15:38.653202 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.653184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 20 10:15:38.653317 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.653199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 20 10:15:38.653317 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.653206    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 20 10:15:38.653317 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.653184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9sptf\""
Jul 20 10:15:38.664795 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.664771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.664882 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.664818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-model-cache\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.664882 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.664852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-dshm\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.664970 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.664885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-home\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.664970 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.664903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.664970 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.664920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpbs\" (UniqueName: \"kubernetes.io/projected/7f57a99f-b84a-4996-91da-ad23d849b146-kube-api-access-lfpbs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.665077 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.664980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f57a99f-b84a-4996-91da-ad23d849b146-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.677167 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.677144    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"]
Jul 20 10:15:38.765664 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.765634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f57a99f-b84a-4996-91da-ad23d849b146-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.765854 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.765670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.765854 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.765700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-model-cache\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.765854 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.765721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-dshm\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.765854 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.765737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-home\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.765854 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.765756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.765854 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.765772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpbs\" (UniqueName: \"kubernetes.io/projected/7f57a99f-b84a-4996-91da-ad23d849b146-kube-api-access-lfpbs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.766161 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.766094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.766161 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.766145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-home\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.766265 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.766206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-model-cache\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.766265 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.766243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-tmp-dir\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.768214 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.768193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-dshm\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.768352 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.768335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f57a99f-b84a-4996-91da-ad23d849b146-tls-certs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.775613 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.775586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpbs\" (UniqueName: \"kubernetes.io/projected/7f57a99f-b84a-4996-91da-ad23d849b146-kube-api-access-lfpbs\") pod \"precise-prefix-cache-test-kserve-66f4d68c65-phdfz\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.946401 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.946307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:38.977486 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.977457    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"]
Jul 20 10:15:38.981351 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.981328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:38.984316 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:38.984298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"precise-prefix-cache-test-epp-sa-dockercfg-k5stv\""
Jul 20 10:15:39.001148 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.001116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"]
Jul 20 10:15:39.068340 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.068309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.068502 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.068358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klwfw\" (UniqueName: \"kubernetes.io/projected/b3b04829-3c0e-41fa-a657-d3903331d9a9-kube-api-access-klwfw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.068502 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.068462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.068502 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.068496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.068669 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.068523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.068669 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.068596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b04829-3c0e-41fa-a657-d3903331d9a9-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.104756 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.104675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"]
Jul 20 10:15:39.107650 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:15:39.107621    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f57a99f_b84a_4996_91da_ad23d849b146.slice/crio-81e95141a5495cb89f57d8019c075c4693adc653064352e6c7889cf2a3b62398 WatchSource:0}: Error finding container 81e95141a5495cb89f57d8019c075c4693adc653064352e6c7889cf2a3b62398: Status 404 returned error can't find the container with id 81e95141a5495cb89f57d8019c075c4693adc653064352e6c7889cf2a3b62398
Jul 20 10:15:39.162044 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.162010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" event={"ID":"7f57a99f-b84a-4996-91da-ad23d849b146","Type":"ContainerStarted","Data":"81e95141a5495cb89f57d8019c075c4693adc653064352e6c7889cf2a3b62398"}
Jul 20 10:15:39.169647 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.169621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.169735 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.169666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klwfw\" (UniqueName: \"kubernetes.io/projected/b3b04829-3c0e-41fa-a657-d3903331d9a9-kube-api-access-klwfw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.169735 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.169701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.169735 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.169718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.169888 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.169735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.169888 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.169819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b04829-3c0e-41fa-a657-d3903331d9a9-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.170133 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.170103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.170206 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.170128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.170206 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.170172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.170206 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.170189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.172323 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.172305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b04829-3c0e-41fa-a657-d3903331d9a9-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.187164 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.187144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klwfw\" (UniqueName: \"kubernetes.io/projected/b3b04829-3c0e-41fa-a657-d3903331d9a9-kube-api-access-klwfw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") " pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.291554 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.291505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:15:39.465018 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:39.464966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"]
Jul 20 10:15:39.467208 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:15:39.467182    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b04829_3c0e_41fa_a657_d3903331d9a9.slice/crio-b5ce2a6b64501b2dea728d229fab8a37739377176bed3b426d020a1fc7148743 WatchSource:0}: Error finding container b5ce2a6b64501b2dea728d229fab8a37739377176bed3b426d020a1fc7148743: Status 404 returned error can't find the container with id b5ce2a6b64501b2dea728d229fab8a37739377176bed3b426d020a1fc7148743
Jul 20 10:15:40.172542 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:40.172488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerStarted","Data":"b5ce2a6b64501b2dea728d229fab8a37739377176bed3b426d020a1fc7148743"}
Jul 20 10:15:43.195674 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:43.195572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" event={"ID":"7f57a99f-b84a-4996-91da-ad23d849b146","Type":"ContainerStarted","Data":"4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0"}
Jul 20 10:15:43.197479 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:43.197444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerStarted","Data":"82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a"}
Jul 20 10:15:44.205596 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:44.205515    2577 generic.go:356] "Generic (PLEG): container finished" podID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerID="82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a" exitCode=0
Jul 20 10:15:44.206158 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:44.205601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerDied","Data":"82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a"}
Jul 20 10:15:45.218708 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:45.218669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerStarted","Data":"b28a98b82fd6aa4447b3f7f39df89d4cbf2dc50e01d70b2a3ec856bac8458e3f"}
Jul 20 10:15:47.236026 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:47.235987    2577 generic.go:356] "Generic (PLEG): container finished" podID="7f57a99f-b84a-4996-91da-ad23d849b146" containerID="4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0" exitCode=0
Jul 20 10:15:47.236509 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:47.236062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" event={"ID":"7f57a99f-b84a-4996-91da-ad23d849b146","Type":"ContainerDied","Data":"4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0"}
Jul 20 10:15:48.243853 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:48.243815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" event={"ID":"7f57a99f-b84a-4996-91da-ad23d849b146","Type":"ContainerStarted","Data":"97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160"}
Jul 20 10:15:48.265471 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:48.265420    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" podStartSLOduration=2.851409323 podStartE2EDuration="10.265400808s" totalImagesPullingTime="3.831587789s" totalInitContainerRuntime="3.582403696s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:15:48.264134635 +0000 UTC m=+1158.896400684" watchObservedRunningTime="2026-07-20 10:15:48.265400808 +0000 UTC m=+1158.897666855"
Jul 20 10:15:48.947267 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:48.947232    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:48.947454 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:48.947310    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:48.964351 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:48.964324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:49.266640 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:49.266600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:15:55.292165 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:55.292135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/0.log"
Jul 20 10:15:55.292692 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:55.292573    2577 generic.go:356] "Generic (PLEG): container finished" podID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerID="b28a98b82fd6aa4447b3f7f39df89d4cbf2dc50e01d70b2a3ec856bac8458e3f" exitCode=1
Jul 20 10:15:55.292692 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:15:55.292659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerDied","Data":"b28a98b82fd6aa4447b3f7f39df89d4cbf2dc50e01d70b2a3ec856bac8458e3f"}
Jul 20 10:16:18.433997 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:18.433914    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/0.log"
Jul 20 10:16:18.434424 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:18.434362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerStarted","Data":"a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224"}
Jul 20 10:16:18.434713 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:18.434696    2577 scope.go:117] "RemoveContainer" containerID="b28a98b82fd6aa4447b3f7f39df89d4cbf2dc50e01d70b2a3ec856bac8458e3f"
Jul 20 10:16:19.292017 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.291969    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:19.292204 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.292099    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:19.292204 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.292126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:19.292204 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.292151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:19.441061 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.441035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/0.log"
Jul 20 10:16:19.441513 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.441457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerStarted","Data":"07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"}
Jul 20 10:16:19.441759 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.441741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:19.469548 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:19.469480    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podStartSLOduration=3.610311925 podStartE2EDuration="41.469462923s" totalImagesPullingTime="37.189751726s" totalInitContainerRuntime="669.399272ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:15:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:16:19.468388751 +0000 UTC m=+1190.100654803" watchObservedRunningTime="2026-07-20 10:16:19.469462923 +0000 UTC m=+1190.101728970"
Jul 20 10:16:29.293782 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:16:29.293747    2577 logging.go:55] [core] [Channel #74 SubChannel #75]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.42:9003", ServerName: "10.133.0.42:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.42:9003: connect: connection refused"
Jul 20 10:16:29.295176 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.295153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:29.296315 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.296293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:29.498468 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.498433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:16:29.499270 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.499249    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/0.log"
Jul 20 10:16:29.499659 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.499629    2577 generic.go:356] "Generic (PLEG): container finished" podID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerID="07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae" exitCode=1
Jul 20 10:16:29.499768 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.499714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerDied","Data":"07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"}
Jul 20 10:16:29.499817 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.499778    2577 scope.go:117] "RemoveContainer" containerID="b28a98b82fd6aa4447b3f7f39df89d4cbf2dc50e01d70b2a3ec856bac8458e3f"
Jul 20 10:16:29.500089 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.500060    2577 scope.go:117] "RemoveContainer" containerID="07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"
Jul 20 10:16:29.500336 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:16:29.500318    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_e2e-test-llm-inference-service-312caf2d(b3b04829-3c0e-41fa-a657-d3903331d9a9)\"" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9"
Jul 20 10:16:29.923823 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.923790    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:16:29.924668 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.924643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:16:29.945283 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.945249    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:16:29.946453 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:29.946431    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:16:30.293815 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:30.293768    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.42:9003\" within 1s: context deadline exceeded"
Jul 20 10:16:30.506101 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:30.506074    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:16:30.507176 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:30.507151    2577 scope.go:117] "RemoveContainer" containerID="07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"
Jul 20 10:16:30.507389 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:16:30.507370    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_e2e-test-llm-inference-service-312caf2d(b3b04829-3c0e-41fa-a657-d3903331d9a9)\"" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9"
Jul 20 10:16:39.292313 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:39.292282    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:39.292927 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:39.292815    2577 scope.go:117] "RemoveContainer" containerID="07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"
Jul 20 10:16:39.553854 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:39.553775    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:16:39.554597 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:39.554574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerStarted","Data":"73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183"}
Jul 20 10:16:39.554917 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:39.554899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:16:49.293194 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:16:49.293157    2577 logging.go:55] [core] [Channel #76 SubChannel #77]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.42:9003", ServerName: "10.133.0.42:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.42:9003: connect: connection refused"
Jul 20 10:16:50.292935 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:16:50.292886    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.42:9003\" within 1s: context deadline exceeded"
Jul 20 10:17:10.560944 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:10.560913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:17:11.887111 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:11.887077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"]
Jul 20 10:17:11.889657 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:11.889446    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"]
Jul 20 10:17:11.889758 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:11.889685    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="tokenizer" containerID="cri-o://a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224" gracePeriod=60
Jul 20 10:17:11.899165 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:11.899129    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:17:11.899165 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:17:11.899165 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:17:11.899165 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160" cmd=["/bin/sleep","15"]
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:11.899184    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" message="" err=<
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:11.899200    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="main" containerID="cri-o://97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160"
Jul 20 10:17:11.899462 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:11.899245    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="main" containerID="cri-o://97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160" gracePeriod=60
Jul 20 10:17:12.275726 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.275698    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:17:12.364780 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.364751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-home\") pod \"7f57a99f-b84a-4996-91da-ad23d849b146\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") "
Jul 20 10:17:12.364977 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.364795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f57a99f-b84a-4996-91da-ad23d849b146-tls-certs\") pod \"7f57a99f-b84a-4996-91da-ad23d849b146\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") "
Jul 20 10:17:12.364977 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.364818    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-tmp-dir\") pod \"7f57a99f-b84a-4996-91da-ad23d849b146\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") "
Jul 20 10:17:12.364977 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.364863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-kserve-provision-location\") pod \"7f57a99f-b84a-4996-91da-ad23d849b146\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") "
Jul 20 10:17:12.364977 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.364889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-dshm\") pod \"7f57a99f-b84a-4996-91da-ad23d849b146\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") "
Jul 20 10:17:12.364977 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.364919    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfpbs\" (UniqueName: \"kubernetes.io/projected/7f57a99f-b84a-4996-91da-ad23d849b146-kube-api-access-lfpbs\") pod \"7f57a99f-b84a-4996-91da-ad23d849b146\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") "
Jul 20 10:17:12.365250 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.364980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-model-cache\") pod \"7f57a99f-b84a-4996-91da-ad23d849b146\" (UID: \"7f57a99f-b84a-4996-91da-ad23d849b146\") "
Jul 20 10:17:12.365250 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.365073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-home" (OuterVolumeSpecName: "home") pod "7f57a99f-b84a-4996-91da-ad23d849b146" (UID: "7f57a99f-b84a-4996-91da-ad23d849b146"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.365250 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.365183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f57a99f-b84a-4996-91da-ad23d849b146" (UID: "7f57a99f-b84a-4996-91da-ad23d849b146"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.365406 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.365287    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.365406 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.365306    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.365406 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.365339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f57a99f-b84a-4996-91da-ad23d849b146" (UID: "7f57a99f-b84a-4996-91da-ad23d849b146"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.367355 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.367328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-dshm" (OuterVolumeSpecName: "dshm") pod "7f57a99f-b84a-4996-91da-ad23d849b146" (UID: "7f57a99f-b84a-4996-91da-ad23d849b146"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.367480 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.367437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f57a99f-b84a-4996-91da-ad23d849b146-kube-api-access-lfpbs" (OuterVolumeSpecName: "kube-api-access-lfpbs") pod "7f57a99f-b84a-4996-91da-ad23d849b146" (UID: "7f57a99f-b84a-4996-91da-ad23d849b146"). InnerVolumeSpecName "kube-api-access-lfpbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:17:12.367566 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.367516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f57a99f-b84a-4996-91da-ad23d849b146-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f57a99f-b84a-4996-91da-ad23d849b146" (UID: "7f57a99f-b84a-4996-91da-ad23d849b146"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:17:12.423078 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.422984    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f57a99f-b84a-4996-91da-ad23d849b146" (UID: "7f57a99f-b84a-4996-91da-ad23d849b146"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:12.465809 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.465780    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.465809 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.465807    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f57a99f-b84a-4996-91da-ad23d849b146-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.466012 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.465817    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-kserve-provision-location\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.466012 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.465826    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f57a99f-b84a-4996-91da-ad23d849b146-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.466012 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.465835    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpbs\" (UniqueName: \"kubernetes.io/projected/7f57a99f-b84a-4996-91da-ad23d849b146-kube-api-access-lfpbs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:12.727495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.727416    2577 generic.go:356] "Generic (PLEG): container finished" podID="7f57a99f-b84a-4996-91da-ad23d849b146" containerID="97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160" exitCode=0
Jul 20 10:17:12.727495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.727465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" event={"ID":"7f57a99f-b84a-4996-91da-ad23d849b146","Type":"ContainerDied","Data":"97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160"}
Jul 20 10:17:12.727709 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.727503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" event={"ID":"7f57a99f-b84a-4996-91da-ad23d849b146","Type":"ContainerDied","Data":"81e95141a5495cb89f57d8019c075c4693adc653064352e6c7889cf2a3b62398"}
Jul 20 10:17:12.727709 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.727524    2577 scope.go:117] "RemoveContainer" containerID="97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160"
Jul 20 10:17:12.727709 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.727467    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"
Jul 20 10:17:12.738094 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.738075    2577 scope.go:117] "RemoveContainer" containerID="4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0"
Jul 20 10:17:12.746505 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.746478    2577 scope.go:117] "RemoveContainer" containerID="97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160"
Jul 20 10:17:12.746833 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:12.746811    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160\": container with ID starting with 97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160 not found: ID does not exist" containerID="97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160"
Jul 20 10:17:12.746929 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.746842    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160"} err="failed to get container status \"97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160\": rpc error: code = NotFound desc = could not find container \"97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160\": container with ID starting with 97d756a09cc667bfebf123e25a7abe15c0dd07edf612770cb52fac67ab64f160 not found: ID does not exist"
Jul 20 10:17:12.746929 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.746860    2577 scope.go:117] "RemoveContainer" containerID="4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0"
Jul 20 10:17:12.747124 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:12.747105    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0\": container with ID starting with 4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0 not found: ID does not exist" containerID="4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0"
Jul 20 10:17:12.747184 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.747130    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0"} err="failed to get container status \"4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0\": rpc error: code = NotFound desc = could not find container \"4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0\": container with ID starting with 4f894d4a72c76674c01bef4f8937fcd4118890b656943e16d24fe568a0ce93a0 not found: ID does not exist"
Jul 20 10:17:12.756112 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.756071    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"]
Jul 20 10:17:12.762302 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:12.762271    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz"]
Jul 20 10:17:13.249630 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:13.249589    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-66f4d68c65-phdfz" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 20 10:17:13.735047 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:13.735019    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:17:13.735687 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:13.735667    2577 generic.go:356] "Generic (PLEG): container finished" podID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerID="a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224" exitCode=0
Jul 20 10:17:13.735760 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:13.735743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerDied","Data":"a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224"}
Jul 20 10:17:13.796717 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:13.796675    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" path="/var/lib/kubelet/pods/7f57a99f-b84a-4996-91da-ad23d849b146/volumes"
Jul 20 10:17:19.296054 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:19.296015    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="tokenizer" probeResult="failure" output="Get \"http://10.133.0.42:8082/healthz\": dial tcp 10.133.0.42:8082: connect: connection refused"
Jul 20 10:17:20.561403 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:20.561365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 20 10:17:26.916145 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:26.916096    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main" containerID="cri-o://73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183" gracePeriod=45
Jul 20 10:17:27.164347 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.164324    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:17:27.167310 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.167260    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:17:27.302389 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-cache\") pod \"b3b04829-3c0e-41fa-a657-d3903331d9a9\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") "
Jul 20 10:17:27.302389 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-tmp\") pod \"b3b04829-3c0e-41fa-a657-d3903331d9a9\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") "
Jul 20 10:17:27.302643 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302401    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-uds\") pod \"b3b04829-3c0e-41fa-a657-d3903331d9a9\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") "
Jul 20 10:17:27.302643 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-kserve-provision-location\") pod \"b3b04829-3c0e-41fa-a657-d3903331d9a9\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") "
Jul 20 10:17:27.302643 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302563    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klwfw\" (UniqueName: \"kubernetes.io/projected/b3b04829-3c0e-41fa-a657-d3903331d9a9-kube-api-access-klwfw\") pod \"b3b04829-3c0e-41fa-a657-d3903331d9a9\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") "
Jul 20 10:17:27.302643 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b04829-3c0e-41fa-a657-d3903331d9a9-tls-certs\") pod \"b3b04829-3c0e-41fa-a657-d3903331d9a9\" (UID: \"b3b04829-3c0e-41fa-a657-d3903331d9a9\") "
Jul 20 10:17:27.302862 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b3b04829-3c0e-41fa-a657-d3903331d9a9" (UID: "b3b04829-3c0e-41fa-a657-d3903331d9a9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:27.302862 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b3b04829-3c0e-41fa-a657-d3903331d9a9" (UID: "b3b04829-3c0e-41fa-a657-d3903331d9a9"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:27.302973 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b3b04829-3c0e-41fa-a657-d3903331d9a9" (UID: "b3b04829-3c0e-41fa-a657-d3903331d9a9"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:27.302973 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302940    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:27.302973 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302960    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-tmp\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:27.303131 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.302974    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-tokenizer-uds\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:27.303350 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.303327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3b04829-3c0e-41fa-a657-d3903331d9a9" (UID: "b3b04829-3c0e-41fa-a657-d3903331d9a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:17:27.304954 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.304928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b04829-3c0e-41fa-a657-d3903331d9a9-kube-api-access-klwfw" (OuterVolumeSpecName: "kube-api-access-klwfw") pod "b3b04829-3c0e-41fa-a657-d3903331d9a9" (UID: "b3b04829-3c0e-41fa-a657-d3903331d9a9"). InnerVolumeSpecName "kube-api-access-klwfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:17:27.304954 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.304935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b04829-3c0e-41fa-a657-d3903331d9a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3b04829-3c0e-41fa-a657-d3903331d9a9" (UID: "b3b04829-3c0e-41fa-a657-d3903331d9a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:17:27.403507 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.403470    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b04829-3c0e-41fa-a657-d3903331d9a9-kserve-provision-location\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:27.403507 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.403502    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klwfw\" (UniqueName: \"kubernetes.io/projected/b3b04829-3c0e-41fa-a657-d3903331d9a9-kube-api-access-klwfw\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:27.403507 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.403513    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3b04829-3c0e-41fa-a657-d3903331d9a9-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:17:27.817357 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.817331    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x_b3b04829-3c0e-41fa-a657-d3903331d9a9/main/1.log"
Jul 20 10:17:27.817995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.817975    2577 generic.go:356] "Generic (PLEG): container finished" podID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerID="73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183" exitCode=0
Jul 20 10:17:27.818075 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.818058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"
Jul 20 10:17:27.818136 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.818064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerDied","Data":"73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183"}
Jul 20 10:17:27.818192 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.818168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x" event={"ID":"b3b04829-3c0e-41fa-a657-d3903331d9a9","Type":"ContainerDied","Data":"b5ce2a6b64501b2dea728d229fab8a37739377176bed3b426d020a1fc7148743"}
Jul 20 10:17:27.818250 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.818209    2577 scope.go:117] "RemoveContainer" containerID="73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183"
Jul 20 10:17:27.827589 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.827576    2577 scope.go:117] "RemoveContainer" containerID="07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"
Jul 20 10:17:27.835413 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.835398    2577 scope.go:117] "RemoveContainer" containerID="a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224"
Jul 20 10:17:27.838218 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.838194    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"]
Jul 20 10:17:27.841839 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.841815    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/precise-prefix-cache-test-kserve-router-scheduler-685f794d7kg2x"]
Jul 20 10:17:27.844436 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.844422    2577 scope.go:117] "RemoveContainer" containerID="82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a"
Jul 20 10:17:27.852052 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.852032    2577 scope.go:117] "RemoveContainer" containerID="73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183"
Jul 20 10:17:27.852330 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:27.852310    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183\": container with ID starting with 73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183 not found: ID does not exist" containerID="73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183"
Jul 20 10:17:27.852374 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.852341    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183"} err="failed to get container status \"73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183\": rpc error: code = NotFound desc = could not find container \"73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183\": container with ID starting with 73cf0958b83247a8d8191a9c4f05021673c8f6e5c2e068c96c50fe2c4376e183 not found: ID does not exist"
Jul 20 10:17:27.852374 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.852360    2577 scope.go:117] "RemoveContainer" containerID="07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"
Jul 20 10:17:27.852654 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:27.852634    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae\": container with ID starting with 07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae not found: ID does not exist" containerID="07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"
Jul 20 10:17:27.852721 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.852660    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae"} err="failed to get container status \"07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae\": rpc error: code = NotFound desc = could not find container \"07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae\": container with ID starting with 07897ac5364ebfa7b7f11f18ea8f009e8fa212a567c837c3f8b9ac2878ec57ae not found: ID does not exist"
Jul 20 10:17:27.852721 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.852676    2577 scope.go:117] "RemoveContainer" containerID="a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224"
Jul 20 10:17:27.852873 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:27.852856    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224\": container with ID starting with a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224 not found: ID does not exist" containerID="a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224"
Jul 20 10:17:27.852914 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.852877    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224"} err="failed to get container status \"a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224\": rpc error: code = NotFound desc = could not find container \"a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224\": container with ID starting with a4237df4bd00d424bfa76dae58d1f57e84c0281a615e4d00086dce010e743224 not found: ID does not exist"
Jul 20 10:17:27.852914 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.852891    2577 scope.go:117] "RemoveContainer" containerID="82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a"
Jul 20 10:17:27.853107 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:17:27.853092    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a\": container with ID starting with 82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a not found: ID does not exist" containerID="82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a"
Jul 20 10:17:27.853154 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:27.853111    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a"} err="failed to get container status \"82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a\": rpc error: code = NotFound desc = could not find container \"82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a\": container with ID starting with 82256b3d6f5fd380b5cc808aae0327be5978c49f6a0eb39a2509534b43a0348a not found: ID does not exist"
Jul 20 10:17:29.796109 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:29.796074    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" path="/var/lib/kubelet/pods/b3b04829-3c0e-41fa-a657-d3903331d9a9/volumes"
Jul 20 10:17:41.503859 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.503824    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"]
Jul 20 10:17:41.504209 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504194    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="storage-initializer"
Jul 20 10:17:41.504209 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504205    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="storage-initializer"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504216    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504222    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504233    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="main"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="main"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504246    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="storage-initializer"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504251    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="storage-initializer"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504263    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504269    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504282    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504286    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504601 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504298    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="tokenizer"
Jul 20 10:17:41.504601 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504303    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="tokenizer"
Jul 20 10:17:41.504601 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504357    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f57a99f-b84a-4996-91da-ad23d849b146" containerName="main"
Jul 20 10:17:41.504601 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504367    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504601 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504372    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.504601 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504378    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="tokenizer"
Jul 20 10:17:41.504601 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.504383    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b04829-3c0e-41fa-a657-d3903331d9a9" containerName="main"
Jul 20 10:17:41.509075 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.509056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.511829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.511805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 20 10:17:41.512434 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.512420    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-m5lgw\""
Jul 20 10:17:41.512544 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.512420    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 20 10:17:41.512544 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.512420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 20 10:17:41.518543 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.518508    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"]
Jul 20 10:17:41.634829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.634795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.634829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.634831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.635132 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.634883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97af1997-f986-4179-9e8a-181a973f5aed-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.635132 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.635021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.635132 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.635049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.635132 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.635072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9l5\" (UniqueName: \"kubernetes.io/projected/97af1997-f986-4179-9e8a-181a973f5aed-kube-api-access-9s9l5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736277 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736277 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736701 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9l5\" (UniqueName: \"kubernetes.io/projected/97af1997-f986-4179-9e8a-181a973f5aed-kube-api-access-9s9l5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736701 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736701 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736701 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97af1997-f986-4179-9e8a-181a973f5aed-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736898 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.736898 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.736835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.737101 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.737071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.739314 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.739285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.739435 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.739406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97af1997-f986-4179-9e8a-181a973f5aed-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.747066 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.747043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9l5\" (UniqueName: \"kubernetes.io/projected/97af1997-f986-4179-9e8a-181a973f5aed-kube-api-access-9s9l5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.819698 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.819613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:41.973306 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:41.973269    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"]
Jul 20 10:17:41.976802 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:17:41.976769    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97af1997_f986_4179_9e8a_181a973f5aed.slice/crio-de60716217a70d3c39ff7ac96bc7c5a4e9783c3979ef995737378e77d3764149 WatchSource:0}: Error finding container de60716217a70d3c39ff7ac96bc7c5a4e9783c3979ef995737378e77d3764149: Status 404 returned error can't find the container with id de60716217a70d3c39ff7ac96bc7c5a4e9783c3979ef995737378e77d3764149
Jul 20 10:17:42.891355 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:42.891323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" event={"ID":"97af1997-f986-4179-9e8a-181a973f5aed","Type":"ContainerStarted","Data":"516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35"}
Jul 20 10:17:42.891773 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:42.891375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" event={"ID":"97af1997-f986-4179-9e8a-181a973f5aed","Type":"ContainerStarted","Data":"de60716217a70d3c39ff7ac96bc7c5a4e9783c3979ef995737378e77d3764149"}
Jul 20 10:17:42.910524 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:42.910474    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" podStartSLOduration=1.910459675 podStartE2EDuration="1.910459675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:17:42.908272572 +0000 UTC m=+1273.540538618" watchObservedRunningTime="2026-07-20 10:17:42.910459675 +0000 UTC m=+1273.542725723"
Jul 20 10:17:43.820704 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:43.820674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:51.820355 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:51.820317    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:51.832900 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:51.832875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:17:51.945394 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:17:51.945365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:18:15.431275 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.431240    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"]
Jul 20 10:18:15.440845 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:18:15.440802    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:18:15.440845 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:18:15.440845 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:18:15.440845 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35" cmd=["/bin/sleep","15"]
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.440863    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" message="" err=<
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:18:15.440880    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" podUID="97af1997-f986-4179-9e8a-181a973f5aed" containerName="main" containerID="cri-o://516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35"
Jul 20 10:18:15.441079 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.440919    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" podUID="97af1997-f986-4179-9e8a-181a973f5aed" containerName="main" containerID="cri-o://516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35" gracePeriod=60
Jul 20 10:18:15.794150 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.794126    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:18:15.947304 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-tmp-dir\") pod \"97af1997-f986-4179-9e8a-181a973f5aed\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") "
Jul 20 10:18:15.947495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-home\") pod \"97af1997-f986-4179-9e8a-181a973f5aed\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") "
Jul 20 10:18:15.947495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-model-cache\") pod \"97af1997-f986-4179-9e8a-181a973f5aed\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") "
Jul 20 10:18:15.947495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97af1997-f986-4179-9e8a-181a973f5aed-tls-certs\") pod \"97af1997-f986-4179-9e8a-181a973f5aed\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") "
Jul 20 10:18:15.947495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-dshm\") pod \"97af1997-f986-4179-9e8a-181a973f5aed\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") "
Jul 20 10:18:15.947764 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s9l5\" (UniqueName: \"kubernetes.io/projected/97af1997-f986-4179-9e8a-181a973f5aed-kube-api-access-9s9l5\") pod \"97af1997-f986-4179-9e8a-181a973f5aed\" (UID: \"97af1997-f986-4179-9e8a-181a973f5aed\") "
Jul 20 10:18:15.947764 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97af1997-f986-4179-9e8a-181a973f5aed" (UID: "97af1997-f986-4179-9e8a-181a973f5aed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:15.947764 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-home" (OuterVolumeSpecName: "home") pod "97af1997-f986-4179-9e8a-181a973f5aed" (UID: "97af1997-f986-4179-9e8a-181a973f5aed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:15.947764 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947715    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-model-cache" (OuterVolumeSpecName: "model-cache") pod "97af1997-f986-4179-9e8a-181a973f5aed" (UID: "97af1997-f986-4179-9e8a-181a973f5aed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:15.947959 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947910    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:18:15.947959 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947929    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:18:15.947959 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.947943    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:18:15.949934 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.949909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97af1997-f986-4179-9e8a-181a973f5aed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97af1997-f986-4179-9e8a-181a973f5aed" (UID: "97af1997-f986-4179-9e8a-181a973f5aed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:18:15.950231 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.950204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97af1997-f986-4179-9e8a-181a973f5aed-kube-api-access-9s9l5" (OuterVolumeSpecName: "kube-api-access-9s9l5") pod "97af1997-f986-4179-9e8a-181a973f5aed" (UID: "97af1997-f986-4179-9e8a-181a973f5aed"). InnerVolumeSpecName "kube-api-access-9s9l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:18:15.950231 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:15.950199    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-dshm" (OuterVolumeSpecName: "dshm") pod "97af1997-f986-4179-9e8a-181a973f5aed" (UID: "97af1997-f986-4179-9e8a-181a973f5aed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:18:16.048948 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.048858    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9l5\" (UniqueName: \"kubernetes.io/projected/97af1997-f986-4179-9e8a-181a973f5aed-kube-api-access-9s9l5\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:18:16.048948 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.048891    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97af1997-f986-4179-9e8a-181a973f5aed-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:18:16.048948 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.048902    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97af1997-f986-4179-9e8a-181a973f5aed-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:18:16.056966 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.056936    2577 generic.go:356] "Generic (PLEG): container finished" podID="97af1997-f986-4179-9e8a-181a973f5aed" containerID="516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35" exitCode=0
Jul 20 10:18:16.057130 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.056985    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"
Jul 20 10:18:16.057130 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.057024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" event={"ID":"97af1997-f986-4179-9e8a-181a973f5aed","Type":"ContainerDied","Data":"516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35"}
Jul 20 10:18:16.057130 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.057089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr" event={"ID":"97af1997-f986-4179-9e8a-181a973f5aed","Type":"ContainerDied","Data":"de60716217a70d3c39ff7ac96bc7c5a4e9783c3979ef995737378e77d3764149"}
Jul 20 10:18:16.057130 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.057118    2577 scope.go:117] "RemoveContainer" containerID="516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35"
Jul 20 10:18:16.066470 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.066447    2577 scope.go:117] "RemoveContainer" containerID="516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35"
Jul 20 10:18:16.066765 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:18:16.066746    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35\": container with ID starting with 516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35 not found: ID does not exist" containerID="516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35"
Jul 20 10:18:16.066820 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.066774    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35"} err="failed to get container status \"516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35\": rpc error: code = NotFound desc = could not find container \"516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35\": container with ID starting with 516fb25da7414646bb57fd12c31de3ba41e223d1581f3669a535993786c42e35 not found: ID does not exist"
Jul 20 10:18:16.077825 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.077802    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"]
Jul 20 10:18:16.083291 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:16.083249    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fm2dr"]
Jul 20 10:18:17.796910 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:17.796875    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97af1997-f986-4179-9e8a-181a973f5aed" path="/var/lib/kubelet/pods/97af1997-f986-4179-9e8a-181a973f5aed/volumes"
Jul 20 10:18:38.893765 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.893723    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"]
Jul 20 10:18:38.894175 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.894132    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97af1997-f986-4179-9e8a-181a973f5aed" containerName="main"
Jul 20 10:18:38.894175 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.894143    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97af1997-f986-4179-9e8a-181a973f5aed" containerName="main"
Jul 20 10:18:38.894247 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.894218    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97af1997-f986-4179-9e8a-181a973f5aed" containerName="main"
Jul 20 10:18:38.898209 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.898192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:38.901804 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.901783    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 20 10:18:38.903500 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.903480    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-mx7hf\""
Jul 20 10:18:38.912753 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.912734    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 20 10:18:38.912874 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.912766    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 20 10:18:38.919256 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.919235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"]
Jul 20 10:18:38.965335 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.965302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:38.965335 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.965340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c41dac-ce00-4c18-b631-d8f70d074c57-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:38.965335 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.965365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smk5\" (UniqueName: \"kubernetes.io/projected/b4c41dac-ce00-4c18-b631-d8f70d074c57-kube-api-access-8smk5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:38.965755 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.965494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:38.965755 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.965521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:38.965755 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:38.965561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066044 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smk5\" (UniqueName: \"kubernetes.io/projected/b4c41dac-ce00-4c18-b631-d8f70d074c57-kube-api-access-8smk5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c41dac-ce00-4c18-b631-d8f70d074c57-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066598 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066714 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.066714 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.066692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.068429 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.068408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.068727 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.068711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c41dac-ce00-4c18-b631-d8f70d074c57-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.077924 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.077901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smk5\" (UniqueName: \"kubernetes.io/projected/b4c41dac-ce00-4c18-b631-d8f70d074c57-kube-api-access-8smk5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.208716 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.208673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:39.356409 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:39.356225    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"]
Jul 20 10:18:39.358686 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:18:39.358658    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c41dac_ce00_4c18_b631_d8f70d074c57.slice/crio-f82caa4360443a3da9db8a861776cad481e21e24906ce7c483abe2227aca7f57 WatchSource:0}: Error finding container f82caa4360443a3da9db8a861776cad481e21e24906ce7c483abe2227aca7f57: Status 404 returned error can't find the container with id f82caa4360443a3da9db8a861776cad481e21e24906ce7c483abe2227aca7f57
Jul 20 10:18:40.174048 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:40.174016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" event={"ID":"b4c41dac-ce00-4c18-b631-d8f70d074c57","Type":"ContainerStarted","Data":"2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7"}
Jul 20 10:18:40.174421 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:40.174064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" event={"ID":"b4c41dac-ce00-4c18-b631-d8f70d074c57","Type":"ContainerStarted","Data":"f82caa4360443a3da9db8a861776cad481e21e24906ce7c483abe2227aca7f57"}
Jul 20 10:18:40.192696 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:40.192636    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" podStartSLOduration=2.192602834 podStartE2EDuration="2.192602834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:18:40.192198026 +0000 UTC m=+1330.824464074" watchObservedRunningTime="2026-07-20 10:18:40.192602834 +0000 UTC m=+1330.824868884"
Jul 20 10:18:40.209646 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:40.209616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:49.209386 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:49.209343    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:49.222050 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:49.222026    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:18:49.233422 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:18:49.233400    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:20:30.334337 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.334295    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"]
Jul 20 10:20:30.343594 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:20:30.343557    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:20:30.343594 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:20:30.343594 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:20:30.343594 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7" cmd=["/bin/sleep","15"]
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.343615    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" message="" err=<
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:20:30.343632    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" podUID="b4c41dac-ce00-4c18-b631-d8f70d074c57" containerName="main" containerID="cri-o://2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7"
Jul 20 10:20:30.343917 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.343677    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" podUID="b4c41dac-ce00-4c18-b631-d8f70d074c57" containerName="main" containerID="cri-o://2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7" gracePeriod=60
Jul 20 10:20:30.711222 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.711193    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:20:30.726782 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.726756    2577 generic.go:356] "Generic (PLEG): container finished" podID="b4c41dac-ce00-4c18-b631-d8f70d074c57" containerID="2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7" exitCode=0
Jul 20 10:20:30.726928 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.726802    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"
Jul 20 10:20:30.726928 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.726843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" event={"ID":"b4c41dac-ce00-4c18-b631-d8f70d074c57","Type":"ContainerDied","Data":"2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7"}
Jul 20 10:20:30.726928 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.726901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr" event={"ID":"b4c41dac-ce00-4c18-b631-d8f70d074c57","Type":"ContainerDied","Data":"f82caa4360443a3da9db8a861776cad481e21e24906ce7c483abe2227aca7f57"}
Jul 20 10:20:30.727032 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.726930    2577 scope.go:117] "RemoveContainer" containerID="2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7"
Jul 20 10:20:30.736777 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.736757    2577 scope.go:117] "RemoveContainer" containerID="2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7"
Jul 20 10:20:30.737070 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:20:30.737051    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7\": container with ID starting with 2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7 not found: ID does not exist" containerID="2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7"
Jul 20 10:20:30.737123 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.737084    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7"} err="failed to get container status \"2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7\": rpc error: code = NotFound desc = could not find container \"2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7\": container with ID starting with 2848e008f75b4ca9f25cc212b1457c633441fb116c7ce238dbc27d8256785fa7 not found: ID does not exist"
Jul 20 10:20:30.812595 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.812554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-model-cache\") pod \"b4c41dac-ce00-4c18-b631-d8f70d074c57\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") "
Jul 20 10:20:30.812797 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.812625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-tmp-dir\") pod \"b4c41dac-ce00-4c18-b631-d8f70d074c57\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") "
Jul 20 10:20:30.812797 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.812651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c41dac-ce00-4c18-b631-d8f70d074c57-tls-certs\") pod \"b4c41dac-ce00-4c18-b631-d8f70d074c57\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") "
Jul 20 10:20:30.812797 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.812686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8smk5\" (UniqueName: \"kubernetes.io/projected/b4c41dac-ce00-4c18-b631-d8f70d074c57-kube-api-access-8smk5\") pod \"b4c41dac-ce00-4c18-b631-d8f70d074c57\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") "
Jul 20 10:20:30.812797 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.812713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-home\") pod \"b4c41dac-ce00-4c18-b631-d8f70d074c57\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") "
Jul 20 10:20:30.813019 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.812807    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-dshm\") pod \"b4c41dac-ce00-4c18-b631-d8f70d074c57\" (UID: \"b4c41dac-ce00-4c18-b631-d8f70d074c57\") "
Jul 20 10:20:30.813019 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.812836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4c41dac-ce00-4c18-b631-d8f70d074c57" (UID: "b4c41dac-ce00-4c18-b631-d8f70d074c57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:30.813117 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.813039    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-home" (OuterVolumeSpecName: "home") pod "b4c41dac-ce00-4c18-b631-d8f70d074c57" (UID: "b4c41dac-ce00-4c18-b631-d8f70d074c57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:30.813117 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.813095    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4c41dac-ce00-4c18-b631-d8f70d074c57" (UID: "b4c41dac-ce00-4c18-b631-d8f70d074c57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:30.813232 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.813139    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:20:30.813232 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.813154    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:20:30.815180 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.815156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c41dac-ce00-4c18-b631-d8f70d074c57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4c41dac-ce00-4c18-b631-d8f70d074c57" (UID: "b4c41dac-ce00-4c18-b631-d8f70d074c57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:30.815288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.815202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-dshm" (OuterVolumeSpecName: "dshm") pod "b4c41dac-ce00-4c18-b631-d8f70d074c57" (UID: "b4c41dac-ce00-4c18-b631-d8f70d074c57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:20:30.815288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.815235    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c41dac-ce00-4c18-b631-d8f70d074c57-kube-api-access-8smk5" (OuterVolumeSpecName: "kube-api-access-8smk5") pod "b4c41dac-ce00-4c18-b631-d8f70d074c57" (UID: "b4c41dac-ce00-4c18-b631-d8f70d074c57"). InnerVolumeSpecName "kube-api-access-8smk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:30.914158 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.914080    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:20:30.914158 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.914107    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c41dac-ce00-4c18-b631-d8f70d074c57-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:20:30.914158 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.914117    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c41dac-ce00-4c18-b631-d8f70d074c57-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:20:30.914158 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:30.914125    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smk5\" (UniqueName: \"kubernetes.io/projected/b4c41dac-ce00-4c18-b631-d8f70d074c57-kube-api-access-8smk5\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:20:31.070760 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:31.070721    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"]
Jul 20 10:20:31.081324 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:31.081289    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-c5gkr"]
Jul 20 10:20:31.795978 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:31.795947    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c41dac-ce00-4c18-b631-d8f70d074c57" path="/var/lib/kubelet/pods/b4c41dac-ce00-4c18-b631-d8f70d074c57/volumes"
Jul 20 10:20:58.904276 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.904232    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"]
Jul 20 10:20:58.905115 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.905090    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c41dac-ce00-4c18-b631-d8f70d074c57" containerName="main"
Jul 20 10:20:58.905210 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.905119    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c41dac-ce00-4c18-b631-d8f70d074c57" containerName="main"
Jul 20 10:20:58.905336 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.905322    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c41dac-ce00-4c18-b631-d8f70d074c57" containerName="main"
Jul 20 10:20:58.910938 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.910913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:58.914281 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.914245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 20 10:20:58.914281 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.914267    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 20 10:20:58.914495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.914249    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 20 10:20:58.914495 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.914461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-hlwqc\""
Jul 20 10:20:58.916519 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:58.916497    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"]
Jul 20 10:20:59.070126 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.070091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.070300 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.070137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfr2d\" (UniqueName: \"kubernetes.io/projected/693ece06-8ee2-4a09-9beb-e78b2de21397-kube-api-access-cfr2d\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.070300 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.070182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.070300 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.070216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.070300 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.070233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.070300 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.070251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693ece06-8ee2-4a09-9beb-e78b2de21397-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171259 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171259 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfr2d\" (UniqueName: \"kubernetes.io/projected/693ece06-8ee2-4a09-9beb-e78b2de21397-kube-api-access-cfr2d\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171259 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171259 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171607 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171607 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693ece06-8ee2-4a09-9beb-e78b2de21397-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171607 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171799 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.171799 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.171786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.173663 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.173636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.173928 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.173912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693ece06-8ee2-4a09-9beb-e78b2de21397-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.178860 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.178841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfr2d\" (UniqueName: \"kubernetes.io/projected/693ece06-8ee2-4a09-9beb-e78b2de21397-kube-api-access-cfr2d\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.222369 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.222337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:20:59.384066 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.384036    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"]
Jul 20 10:20:59.387498 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:20:59.387472    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693ece06_8ee2_4a09_9beb_e78b2de21397.slice/crio-08d8124486c4224d23fb6b08867a776f54865bd28f4d72c77f4bae9e204d74c6 WatchSource:0}: Error finding container 08d8124486c4224d23fb6b08867a776f54865bd28f4d72c77f4bae9e204d74c6: Status 404 returned error can't find the container with id 08d8124486c4224d23fb6b08867a776f54865bd28f4d72c77f4bae9e204d74c6
Jul 20 10:20:59.871067 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.871027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" event={"ID":"693ece06-8ee2-4a09-9beb-e78b2de21397","Type":"ContainerStarted","Data":"b83993c22d3f4acf7391b8f5c440de056ddc19b357bd7377ae21ee9ce972c343"}
Jul 20 10:20:59.871310 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.871091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" event={"ID":"693ece06-8ee2-4a09-9beb-e78b2de21397","Type":"ContainerStarted","Data":"08d8124486c4224d23fb6b08867a776f54865bd28f4d72c77f4bae9e204d74c6"}
Jul 20 10:20:59.890425 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:20:59.890374    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" podStartSLOduration=1.890358806 podStartE2EDuration="1.890358806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:20:59.888043671 +0000 UTC m=+1470.520309720" watchObservedRunningTime="2026-07-20 10:20:59.890358806 +0000 UTC m=+1470.522624872"
Jul 20 10:21:00.223233 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:21:00.223198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:21:09.222871 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:21:09.222835    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:21:09.235374 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:21:09.235351    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:21:09.935376 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:21:09.935348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:21:29.976974 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:21:29.976944    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:21:29.979048 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:21:29.979028    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:22:43.780603 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:43.780567    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"]
Jul 20 10:22:43.790199 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:22:43.790152    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:22:43.790199 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:22:43.790199 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:22:43.790199 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="b83993c22d3f4acf7391b8f5c440de056ddc19b357bd7377ae21ee9ce972c343" cmd=["/bin/sleep","15"]
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:43.790218    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" message="" err=<
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:22:43.790235    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" podUID="693ece06-8ee2-4a09-9beb-e78b2de21397" containerName="main" containerID="cri-o://b83993c22d3f4acf7391b8f5c440de056ddc19b357bd7377ae21ee9ce972c343"
Jul 20 10:22:43.791029 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:43.790283    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" podUID="693ece06-8ee2-4a09-9beb-e78b2de21397" containerName="main" containerID="cri-o://b83993c22d3f4acf7391b8f5c440de056ddc19b357bd7377ae21ee9ce972c343" gracePeriod=60
Jul 20 10:22:43.924842 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:43.924795    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" podUID="693ece06-8ee2-4a09-9beb-e78b2de21397" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 20 10:22:44.400443 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.400406    2577 generic.go:356] "Generic (PLEG): container finished" podID="693ece06-8ee2-4a09-9beb-e78b2de21397" containerID="b83993c22d3f4acf7391b8f5c440de056ddc19b357bd7377ae21ee9ce972c343" exitCode=0
Jul 20 10:22:44.400721 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.400448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" event={"ID":"693ece06-8ee2-4a09-9beb-e78b2de21397","Type":"ContainerDied","Data":"b83993c22d3f4acf7391b8f5c440de056ddc19b357bd7377ae21ee9ce972c343"}
Jul 20 10:22:44.548780 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.548755    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:22:44.628824 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.628789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-model-cache\") pod \"693ece06-8ee2-4a09-9beb-e78b2de21397\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") "
Jul 20 10:22:44.628995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.628879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-tmp-dir\") pod \"693ece06-8ee2-4a09-9beb-e78b2de21397\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") "
Jul 20 10:22:44.628995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.628908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-dshm\") pod \"693ece06-8ee2-4a09-9beb-e78b2de21397\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") "
Jul 20 10:22:44.628995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.628932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693ece06-8ee2-4a09-9beb-e78b2de21397-tls-certs\") pod \"693ece06-8ee2-4a09-9beb-e78b2de21397\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") "
Jul 20 10:22:44.628995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.628949    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfr2d\" (UniqueName: \"kubernetes.io/projected/693ece06-8ee2-4a09-9beb-e78b2de21397-kube-api-access-cfr2d\") pod \"693ece06-8ee2-4a09-9beb-e78b2de21397\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") "
Jul 20 10:22:44.628995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.628991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-home\") pod \"693ece06-8ee2-4a09-9beb-e78b2de21397\" (UID: \"693ece06-8ee2-4a09-9beb-e78b2de21397\") "
Jul 20 10:22:44.629271 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.629064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-model-cache" (OuterVolumeSpecName: "model-cache") pod "693ece06-8ee2-4a09-9beb-e78b2de21397" (UID: "693ece06-8ee2-4a09-9beb-e78b2de21397"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:44.629271 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.629151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "693ece06-8ee2-4a09-9beb-e78b2de21397" (UID: "693ece06-8ee2-4a09-9beb-e78b2de21397"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:44.629388 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.629330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-home" (OuterVolumeSpecName: "home") pod "693ece06-8ee2-4a09-9beb-e78b2de21397" (UID: "693ece06-8ee2-4a09-9beb-e78b2de21397"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:44.629388 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.629350    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:22:44.629388 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.629366    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:22:44.631348 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.631324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693ece06-8ee2-4a09-9beb-e78b2de21397-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "693ece06-8ee2-4a09-9beb-e78b2de21397" (UID: "693ece06-8ee2-4a09-9beb-e78b2de21397"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:44.631452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.631364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693ece06-8ee2-4a09-9beb-e78b2de21397-kube-api-access-cfr2d" (OuterVolumeSpecName: "kube-api-access-cfr2d") pod "693ece06-8ee2-4a09-9beb-e78b2de21397" (UID: "693ece06-8ee2-4a09-9beb-e78b2de21397"). InnerVolumeSpecName "kube-api-access-cfr2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:44.631452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.631404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-dshm" (OuterVolumeSpecName: "dshm") pod "693ece06-8ee2-4a09-9beb-e78b2de21397" (UID: "693ece06-8ee2-4a09-9beb-e78b2de21397"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:22:44.730185 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.730156    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:22:44.730185 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.730180    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693ece06-8ee2-4a09-9beb-e78b2de21397-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:22:44.730185 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.730191    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfr2d\" (UniqueName: \"kubernetes.io/projected/693ece06-8ee2-4a09-9beb-e78b2de21397-kube-api-access-cfr2d\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:22:44.730412 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:44.730200    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693ece06-8ee2-4a09-9beb-e78b2de21397-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:22:45.407208 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:45.407176    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"
Jul 20 10:22:45.407208 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:45.407181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq" event={"ID":"693ece06-8ee2-4a09-9beb-e78b2de21397","Type":"ContainerDied","Data":"08d8124486c4224d23fb6b08867a776f54865bd28f4d72c77f4bae9e204d74c6"}
Jul 20 10:22:45.407743 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:45.407236    2577 scope.go:117] "RemoveContainer" containerID="b83993c22d3f4acf7391b8f5c440de056ddc19b357bd7377ae21ee9ce972c343"
Jul 20 10:22:45.433038 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:45.433009    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"]
Jul 20 10:22:45.437083 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:45.437063    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969bnttq"]
Jul 20 10:22:45.796323 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:22:45.796290    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="693ece06-8ee2-4a09-9beb-e78b2de21397" path="/var/lib/kubelet/pods/693ece06-8ee2-4a09-9beb-e78b2de21397/volumes"
Jul 20 10:24:21.493541 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.493496    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"]
Jul 20 10:24:21.493958 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.493943    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693ece06-8ee2-4a09-9beb-e78b2de21397" containerName="main"
Jul 20 10:24:21.494006 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.493960    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="693ece06-8ee2-4a09-9beb-e78b2de21397" containerName="main"
Jul 20 10:24:21.494044 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.494025    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="693ece06-8ee2-4a09-9beb-e78b2de21397" containerName="main"
Jul 20 10:24:21.497066 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.497046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.499264 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.499240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 20 10:24:21.499799 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.499778    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 20 10:24:21.499903 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.499801    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-rjn8t\""
Jul 20 10:24:21.499903 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.499823    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 20 10:24:21.506504 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.505444    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"]
Jul 20 10:24:21.620634 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.620602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.620829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.620643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.620829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.620777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.620829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.620816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.620995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.620856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.620995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.620889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdq2\" (UniqueName: \"kubernetes.io/projected/966d3990-67f0-47c7-bbaa-15e7d2f9facc-kube-api-access-nbdq2\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722264 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722264 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722523 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722523 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdq2\" (UniqueName: \"kubernetes.io/projected/966d3990-67f0-47c7-bbaa-15e7d2f9facc-kube-api-access-nbdq2\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722523 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722718 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722718 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722784 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.722818 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.722789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.724913 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.724886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.725293 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.725268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.730562 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.730517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdq2\" (UniqueName: \"kubernetes.io/projected/966d3990-67f0-47c7-bbaa-15e7d2f9facc-kube-api-access-nbdq2\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-747xw\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.809985 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.809904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:21.946179 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:21.946151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"]
Jul 20 10:24:21.948892 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:24:21.948856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966d3990_67f0_47c7_bbaa_15e7d2f9facc.slice/crio-968ad7f41b24b74fc18a3384e07e678974ec6eadbab691d5780c64d849e04b67 WatchSource:0}: Error finding container 968ad7f41b24b74fc18a3384e07e678974ec6eadbab691d5780c64d849e04b67: Status 404 returned error can't find the container with id 968ad7f41b24b74fc18a3384e07e678974ec6eadbab691d5780c64d849e04b67
Jul 20 10:24:22.879616 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:22.879582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" event={"ID":"966d3990-67f0-47c7-bbaa-15e7d2f9facc","Type":"ContainerStarted","Data":"dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060"}
Jul 20 10:24:22.880102 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:22.879663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" event={"ID":"966d3990-67f0-47c7-bbaa-15e7d2f9facc","Type":"ContainerStarted","Data":"968ad7f41b24b74fc18a3384e07e678974ec6eadbab691d5780c64d849e04b67"}
Jul 20 10:24:22.896906 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:22.896809    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" podStartSLOduration=1.8967888290000001 podStartE2EDuration="1.896788829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:24:22.896780453 +0000 UTC m=+1673.529046502" watchObservedRunningTime="2026-07-20 10:24:22.896788829 +0000 UTC m=+1673.529054875"
Jul 20 10:24:23.810466 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:23.810433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:31.810879 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:31.810845    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:31.823400 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:31.823371    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:24:31.940569 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:24:31.940514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:26:30.009283 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:26:30.009250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:26:30.011624 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:26:30.011600    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:31:30.040372 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:31:30.040344    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:31:30.043834 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:31:30.043804    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:36:30.074026 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:36:30.073996    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:36:30.078436 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:36:30.078409    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:39:16.850853 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:16.850816    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"]
Jul 20 10:39:16.860488 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:39:16.860447    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:39:16.860488 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:39:16.860488 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:39:16.860488 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060" cmd=["/bin/sleep","15"]
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:16.860501    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" message="" err=<
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:39:16.860524    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" podUID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" containerName="main" containerID="cri-o://dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060"
Jul 20 10:39:16.860762 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:16.860579    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" podUID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" containerName="main" containerID="cri-o://dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060" gracePeriod=60
Jul 20 10:39:16.930468 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:16.930424    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" podUID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 20 10:39:17.123645 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.123618    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:39:17.138732 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.138701    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-dshm\") pod \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") "
Jul 20 10:39:17.138895 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.138765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbdq2\" (UniqueName: \"kubernetes.io/projected/966d3990-67f0-47c7-bbaa-15e7d2f9facc-kube-api-access-nbdq2\") pod \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") "
Jul 20 10:39:17.138895 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.138811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-model-cache\") pod \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") "
Jul 20 10:39:17.138895 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.138892    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-home\") pod \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") "
Jul 20 10:39:17.139086 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.138918    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tmp-dir\") pod \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") "
Jul 20 10:39:17.139086 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.138956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tls-certs\") pod \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\" (UID: \"966d3990-67f0-47c7-bbaa-15e7d2f9facc\") "
Jul 20 10:39:17.139179 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.139085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-model-cache" (OuterVolumeSpecName: "model-cache") pod "966d3990-67f0-47c7-bbaa-15e7d2f9facc" (UID: "966d3990-67f0-47c7-bbaa-15e7d2f9facc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:17.139179 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.139164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-home" (OuterVolumeSpecName: "home") pod "966d3990-67f0-47c7-bbaa-15e7d2f9facc" (UID: "966d3990-67f0-47c7-bbaa-15e7d2f9facc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:17.139274 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.139181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "966d3990-67f0-47c7-bbaa-15e7d2f9facc" (UID: "966d3990-67f0-47c7-bbaa-15e7d2f9facc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:17.139274 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.139264    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:39:17.139354 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.139282    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:39:17.139354 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.139300    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:39:17.141282 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.141256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-dshm" (OuterVolumeSpecName: "dshm") pod "966d3990-67f0-47c7-bbaa-15e7d2f9facc" (UID: "966d3990-67f0-47c7-bbaa-15e7d2f9facc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:39:17.141407 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.141340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "966d3990-67f0-47c7-bbaa-15e7d2f9facc" (UID: "966d3990-67f0-47c7-bbaa-15e7d2f9facc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:39:17.141407 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.141378    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966d3990-67f0-47c7-bbaa-15e7d2f9facc-kube-api-access-nbdq2" (OuterVolumeSpecName: "kube-api-access-nbdq2") pod "966d3990-67f0-47c7-bbaa-15e7d2f9facc" (UID: "966d3990-67f0-47c7-bbaa-15e7d2f9facc"). InnerVolumeSpecName "kube-api-access-nbdq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:39:17.240200 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.240165    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d3990-67f0-47c7-bbaa-15e7d2f9facc-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:39:17.240200 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.240197    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d3990-67f0-47c7-bbaa-15e7d2f9facc-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:39:17.240200 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.240209    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbdq2\" (UniqueName: \"kubernetes.io/projected/966d3990-67f0-47c7-bbaa-15e7d2f9facc-kube-api-access-nbdq2\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:39:17.370302 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.370208    2577 generic.go:356] "Generic (PLEG): container finished" podID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" containerID="dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060" exitCode=0
Jul 20 10:39:17.370302 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.370260    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"
Jul 20 10:39:17.370506 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.370292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" event={"ID":"966d3990-67f0-47c7-bbaa-15e7d2f9facc","Type":"ContainerDied","Data":"dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060"}
Jul 20 10:39:17.370506 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.370351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw" event={"ID":"966d3990-67f0-47c7-bbaa-15e7d2f9facc","Type":"ContainerDied","Data":"968ad7f41b24b74fc18a3384e07e678974ec6eadbab691d5780c64d849e04b67"}
Jul 20 10:39:17.370506 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.370375    2577 scope.go:117] "RemoveContainer" containerID="dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060"
Jul 20 10:39:17.380998 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.380971    2577 scope.go:117] "RemoveContainer" containerID="dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060"
Jul 20 10:39:17.381318 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:39:17.381294    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060\": container with ID starting with dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060 not found: ID does not exist" containerID="dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060"
Jul 20 10:39:17.381390 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.381335    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060"} err="failed to get container status \"dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060\": rpc error: code = NotFound desc = could not find container \"dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060\": container with ID starting with dd80af112610494e056ea7db9dc520aa1fa5e9d09dc98641f2e3c482a5d05060 not found: ID does not exist"
Jul 20 10:39:17.390452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.390421    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"]
Jul 20 10:39:17.396584 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.396556    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-747xw"]
Jul 20 10:39:17.796751 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:17.796715    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" path="/var/lib/kubelet/pods/966d3990-67f0-47c7-bbaa-15e7d2f9facc/volumes"
Jul 20 10:39:29.407132 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.407097    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"]
Jul 20 10:39:29.407708 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.407689    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" containerName="main"
Jul 20 10:39:29.407797 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.407711    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" containerName="main"
Jul 20 10:39:29.407852 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.407827    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="966d3990-67f0-47c7-bbaa-15e7d2f9facc" containerName="main"
Jul 20 10:39:29.412727 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.412699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.414876 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.414852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-55nbp\""
Jul 20 10:39:29.415288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.415253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 20 10:39:29.415288 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.415279    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 20 10:39:29.415449 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.415279    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 20 10:39:29.421872 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.421844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"]
Jul 20 10:39:29.454564 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.454492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.454782 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.454577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.454782 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.454701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.454782 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.454751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4zx\" (UniqueName: \"kubernetes.io/projected/76fc9373-f122-4776-92c1-17cc389d6522-kube-api-access-jf4zx\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.454908 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.454796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fc9373-f122-4776-92c1-17cc389d6522-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.454908 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.454850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.555829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.555787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.555829 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.555837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4zx\" (UniqueName: \"kubernetes.io/projected/76fc9373-f122-4776-92c1-17cc389d6522-kube-api-access-jf4zx\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.556092 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.555868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fc9373-f122-4776-92c1-17cc389d6522-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.556092 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.555885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.556092 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.555902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.556092 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.555920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.556367 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.556256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.556367 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.556269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.556367 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.556350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.558364 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.558338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.558710 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.558691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fc9373-f122-4776-92c1-17cc389d6522-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.564416 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.564394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4zx\" (UniqueName: \"kubernetes.io/projected/76fc9373-f122-4776-92c1-17cc389d6522-kube-api-access-jf4zx\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.726929 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.726901    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-55nbp\""
Jul 20 10:39:29.735686 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.735645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:29.889712 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:29.889685    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"]
Jul 20 10:39:29.892571 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:39:29.892522    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fc9373_f122_4776_92c1_17cc389d6522.slice/crio-56a6559b6e521db1a77a42cf88dc8e7140cd3a3a9cc72d67a3cf3bdec162f497 WatchSource:0}: Error finding container 56a6559b6e521db1a77a42cf88dc8e7140cd3a3a9cc72d67a3cf3bdec162f497: Status 404 returned error can't find the container with id 56a6559b6e521db1a77a42cf88dc8e7140cd3a3a9cc72d67a3cf3bdec162f497
Jul 20 10:39:30.434791 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:30.434752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" event={"ID":"76fc9373-f122-4776-92c1-17cc389d6522","Type":"ContainerStarted","Data":"14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7"}
Jul 20 10:39:30.435187 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:30.434817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" event={"ID":"76fc9373-f122-4776-92c1-17cc389d6522","Type":"ContainerStarted","Data":"56a6559b6e521db1a77a42cf88dc8e7140cd3a3a9cc72d67a3cf3bdec162f497"}
Jul 20 10:39:30.455217 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:30.455165    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" podStartSLOduration=1.455144502 podStartE2EDuration="1.455144502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:39:30.45500057 +0000 UTC m=+2581.087266619" watchObservedRunningTime="2026-07-20 10:39:30.455144502 +0000 UTC m=+2581.087410551"
Jul 20 10:39:30.736389 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:30.736354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:39.736871 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:39.736830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:39.749630 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:39.749595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:39:40.499330 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:39:40.499300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:41:30.108277 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:41:30.108252    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:41:30.114388 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:41:30.114358    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:46:30.140607 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:46:30.140505    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:46:30.146994 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:46:30.146969    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:51:30.170825 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:51:30.170795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:51:30.178806 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:51:30.178784    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:54:23.017968 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.017936    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"]
Jul 20 10:54:23.027221 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:54:23.027190    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 10:54:23.027221 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:54:23.027221 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:54:23.027221 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7" cmd=["/bin/sleep","15"]
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.027239    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" message="" err=<
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:54:23.027250    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" podUID="76fc9373-f122-4776-92c1-17cc389d6522" containerName="main" containerID="cri-o://14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7"
Jul 20 10:54:23.027452 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.027292    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" podUID="76fc9373-f122-4776-92c1-17cc389d6522" containerName="main" containerID="cri-o://14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7" gracePeriod=60
Jul 20 10:54:23.279491 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.279428    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:54:23.397841 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.397813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-dshm\") pod \"76fc9373-f122-4776-92c1-17cc389d6522\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") "
Jul 20 10:54:23.398010 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.397855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fc9373-f122-4776-92c1-17cc389d6522-tls-certs\") pod \"76fc9373-f122-4776-92c1-17cc389d6522\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") "
Jul 20 10:54:23.398010 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.397886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-model-cache\") pod \"76fc9373-f122-4776-92c1-17cc389d6522\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") "
Jul 20 10:54:23.398010 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.397915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-tmp-dir\") pod \"76fc9373-f122-4776-92c1-17cc389d6522\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") "
Jul 20 10:54:23.398010 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.397974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-home\") pod \"76fc9373-f122-4776-92c1-17cc389d6522\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") "
Jul 20 10:54:23.398192 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.398031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf4zx\" (UniqueName: \"kubernetes.io/projected/76fc9373-f122-4776-92c1-17cc389d6522-kube-api-access-jf4zx\") pod \"76fc9373-f122-4776-92c1-17cc389d6522\" (UID: \"76fc9373-f122-4776-92c1-17cc389d6522\") "
Jul 20 10:54:23.398192 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.398140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-model-cache" (OuterVolumeSpecName: "model-cache") pod "76fc9373-f122-4776-92c1-17cc389d6522" (UID: "76fc9373-f122-4776-92c1-17cc389d6522"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:54:23.398301 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.398204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-home" (OuterVolumeSpecName: "home") pod "76fc9373-f122-4776-92c1-17cc389d6522" (UID: "76fc9373-f122-4776-92c1-17cc389d6522"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:54:23.398301 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.398243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76fc9373-f122-4776-92c1-17cc389d6522" (UID: "76fc9373-f122-4776-92c1-17cc389d6522"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:54:23.398437 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.398330    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:54:23.398437 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.398343    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:54:23.398437 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.398352    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:54:23.400350 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.400322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fc9373-f122-4776-92c1-17cc389d6522-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76fc9373-f122-4776-92c1-17cc389d6522" (UID: "76fc9373-f122-4776-92c1-17cc389d6522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:54:23.400350 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.400331    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-dshm" (OuterVolumeSpecName: "dshm") pod "76fc9373-f122-4776-92c1-17cc389d6522" (UID: "76fc9373-f122-4776-92c1-17cc389d6522"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:54:23.400350 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.400334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fc9373-f122-4776-92c1-17cc389d6522-kube-api-access-jf4zx" (OuterVolumeSpecName: "kube-api-access-jf4zx") pod "76fc9373-f122-4776-92c1-17cc389d6522" (UID: "76fc9373-f122-4776-92c1-17cc389d6522"). InnerVolumeSpecName "kube-api-access-jf4zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:54:23.498995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.498969    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4zx\" (UniqueName: \"kubernetes.io/projected/76fc9373-f122-4776-92c1-17cc389d6522-kube-api-access-jf4zx\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:54:23.498995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.498992    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fc9373-f122-4776-92c1-17cc389d6522-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:54:23.498995 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.499002    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fc9373-f122-4776-92c1-17cc389d6522-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 10:54:23.835857 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.835826    2577 generic.go:356] "Generic (PLEG): container finished" podID="76fc9373-f122-4776-92c1-17cc389d6522" containerID="14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7" exitCode=0
Jul 20 10:54:23.836020 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.835873    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"
Jul 20 10:54:23.836020 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.835877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" event={"ID":"76fc9373-f122-4776-92c1-17cc389d6522","Type":"ContainerDied","Data":"14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7"}
Jul 20 10:54:23.836020 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.835931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8" event={"ID":"76fc9373-f122-4776-92c1-17cc389d6522","Type":"ContainerDied","Data":"56a6559b6e521db1a77a42cf88dc8e7140cd3a3a9cc72d67a3cf3bdec162f497"}
Jul 20 10:54:23.836020 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.835962    2577 scope.go:117] "RemoveContainer" containerID="14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7"
Jul 20 10:54:23.844922 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.844902    2577 scope.go:117] "RemoveContainer" containerID="14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7"
Jul 20 10:54:23.845180 ip-10-0-137-140 kubenswrapper[2577]: E0720 10:54:23.845160    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7\": container with ID starting with 14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7 not found: ID does not exist" containerID="14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7"
Jul 20 10:54:23.845240 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.845187    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7"} err="failed to get container status \"14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7\": rpc error: code = NotFound desc = could not find container \"14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7\": container with ID starting with 14928c68d8500d10933c36cba7634c66821f2f9288edaf97db46e9d24825cfb7 not found: ID does not exist"
Jul 20 10:54:23.857207 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.857181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"]
Jul 20 10:54:23.859905 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:23.859885    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4trw8"]
Jul 20 10:54:25.796608 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:25.796569    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fc9373-f122-4776-92c1-17cc389d6522" path="/var/lib/kubelet/pods/76fc9373-f122-4776-92c1-17cc389d6522/volumes"
Jul 20 10:54:33.153894 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.153862    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 20 10:54:33.154263 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.154252    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fc9373-f122-4776-92c1-17cc389d6522" containerName="main"
Jul 20 10:54:33.154310 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.154266    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fc9373-f122-4776-92c1-17cc389d6522" containerName="main"
Jul 20 10:54:33.154345 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.154322    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fc9373-f122-4776-92c1-17cc389d6522" containerName="main"
Jul 20 10:54:33.159167 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.159145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.164678 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.163312    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 20 10:54:33.164678 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.163616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 20 10:54:33.164678 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.163871    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-8rgxn\""
Jul 20 10:54:33.164678 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.164089    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 20 10:54:33.164678 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.164239    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-vpm2w\""
Jul 20 10:54:33.174612 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.174587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 20 10:54:33.286156 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.286122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.286324 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.286164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4deef5d-09d0-4243-9828-b8713db02ff9-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.286324 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.286223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlw77\" (UniqueName: \"kubernetes.io/projected/b4deef5d-09d0-4243-9828-b8713db02ff9-kube-api-access-wlw77\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.286324 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.286248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.286324 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.286278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.286485 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.286319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387402 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387611 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4deef5d-09d0-4243-9828-b8713db02ff9-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387611 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlw77\" (UniqueName: \"kubernetes.io/projected/b4deef5d-09d0-4243-9828-b8713db02ff9-kube-api-access-wlw77\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387611 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387611 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387611 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387903 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387903 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.387993 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.387943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.390003 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.389980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.390115 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.390043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4deef5d-09d0-4243-9828-b8713db02ff9-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.396852 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.396831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlw77\" (UniqueName: \"kubernetes.io/projected/b4deef5d-09d0-4243-9828-b8713db02ff9-kube-api-access-wlw77\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.478015 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.477991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:33.642387 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.642355    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 20 10:54:33.643593 ip-10-0-137-140 kubenswrapper[2577]: W0720 10:54:33.643569    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4deef5d_09d0_4243_9828_b8713db02ff9.slice/crio-86eb86b8d72ca2302f0da930abf16966525df428e5361d5adc373a1e858e357c WatchSource:0}: Error finding container 86eb86b8d72ca2302f0da930abf16966525df428e5361d5adc373a1e858e357c: Status 404 returned error can't find the container with id 86eb86b8d72ca2302f0da930abf16966525df428e5361d5adc373a1e858e357c
Jul 20 10:54:33.884921 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.884885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"b4deef5d-09d0-4243-9828-b8713db02ff9","Type":"ContainerStarted","Data":"76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b"}
Jul 20 10:54:33.885124 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.884943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"b4deef5d-09d0-4243-9828-b8713db02ff9","Type":"ContainerStarted","Data":"86eb86b8d72ca2302f0da930abf16966525df428e5361d5adc373a1e858e357c"}
Jul 20 10:54:33.906016 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:33.905958    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podStartSLOduration=0.905914575 podStartE2EDuration="905.914575ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:54:33.903857324 +0000 UTC m=+3484.536123378" watchObservedRunningTime="2026-07-20 10:54:33.905914575 +0000 UTC m=+3484.538180623"
Jul 20 10:54:34.478440 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:34.478407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:43.478507 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:43.478463    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:43.491050 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:43.491022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:54:43.944160 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:54:43.944066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 10:56:30.203160 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:56:30.203130    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 10:56:30.211770 ip-10-0-137-140 kubenswrapper[2577]: I0720 10:56:30.211749    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:01:30.235582 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:01:30.235555    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:01:30.245366 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:01:30.245344    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:06:30.267142 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:06:30.267113    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:06:30.279315 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:06:30.279292    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:09:29.339735 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.339700    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 20 11:09:29.349362 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:09:29.349318    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:09:29.349362 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:09:29.349362 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 11:09:29.349362 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b" cmd=["/bin/sleep","15"]
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.349378    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" message="" err=<
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:09:29.349391    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="b4deef5d-09d0-4243-9828-b8713db02ff9" containerName="main" containerID="cri-o://76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b"
Jul 20 11:09:29.349618 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.349434    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="b4deef5d-09d0-4243-9828-b8713db02ff9" containerName="main" containerID="cri-o://76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b" gracePeriod=60
Jul 20 11:09:29.604819 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.604747    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 11:09:29.664622 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-home\") pod \"b4deef5d-09d0-4243-9828-b8713db02ff9\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") "
Jul 20 11:09:29.664842 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664668    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-tmp-dir\") pod \"b4deef5d-09d0-4243-9828-b8713db02ff9\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") "
Jul 20 11:09:29.664842 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664690    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlw77\" (UniqueName: \"kubernetes.io/projected/b4deef5d-09d0-4243-9828-b8713db02ff9-kube-api-access-wlw77\") pod \"b4deef5d-09d0-4243-9828-b8713db02ff9\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") "
Jul 20 11:09:29.664842 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664719    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-dshm\") pod \"b4deef5d-09d0-4243-9828-b8713db02ff9\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") "
Jul 20 11:09:29.664842 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664734    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-model-cache\") pod \"b4deef5d-09d0-4243-9828-b8713db02ff9\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") "
Jul 20 11:09:29.664842 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4deef5d-09d0-4243-9828-b8713db02ff9-tls-certs\") pod \"b4deef5d-09d0-4243-9828-b8713db02ff9\" (UID: \"b4deef5d-09d0-4243-9828-b8713db02ff9\") "
Jul 20 11:09:29.665118 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-home" (OuterVolumeSpecName: "home") pod "b4deef5d-09d0-4243-9828-b8713db02ff9" (UID: "b4deef5d-09d0-4243-9828-b8713db02ff9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:29.665118 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.664987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4deef5d-09d0-4243-9828-b8713db02ff9" (UID: "b4deef5d-09d0-4243-9828-b8713db02ff9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:29.665118 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.665025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4deef5d-09d0-4243-9828-b8713db02ff9" (UID: "b4deef5d-09d0-4243-9828-b8713db02ff9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:29.665118 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.665104    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:09:29.665256 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.665122    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:09:29.665256 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.665132    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:09:29.667206 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.667178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-dshm" (OuterVolumeSpecName: "dshm") pod "b4deef5d-09d0-4243-9828-b8713db02ff9" (UID: "b4deef5d-09d0-4243-9828-b8713db02ff9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:09:29.667417 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.667219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4deef5d-09d0-4243-9828-b8713db02ff9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4deef5d-09d0-4243-9828-b8713db02ff9" (UID: "b4deef5d-09d0-4243-9828-b8713db02ff9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:09:29.667417 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.667233    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4deef5d-09d0-4243-9828-b8713db02ff9-kube-api-access-wlw77" (OuterVolumeSpecName: "kube-api-access-wlw77") pod "b4deef5d-09d0-4243-9828-b8713db02ff9" (UID: "b4deef5d-09d0-4243-9828-b8713db02ff9"). InnerVolumeSpecName "kube-api-access-wlw77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:09:29.766601 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.766561    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4deef5d-09d0-4243-9828-b8713db02ff9-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:09:29.766601 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.766591    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4deef5d-09d0-4243-9828-b8713db02ff9-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:09:29.766601 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:29.766600    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlw77\" (UniqueName: \"kubernetes.io/projected/b4deef5d-09d0-4243-9828-b8713db02ff9-kube-api-access-wlw77\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:09:30.332420 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.332375    2577 generic.go:356] "Generic (PLEG): container finished" podID="b4deef5d-09d0-4243-9828-b8713db02ff9" containerID="76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b" exitCode=0
Jul 20 11:09:30.332664 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.332434    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 20 11:09:30.332664 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.332466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"b4deef5d-09d0-4243-9828-b8713db02ff9","Type":"ContainerDied","Data":"76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b"}
Jul 20 11:09:30.332664 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.332595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"b4deef5d-09d0-4243-9828-b8713db02ff9","Type":"ContainerDied","Data":"86eb86b8d72ca2302f0da930abf16966525df428e5361d5adc373a1e858e357c"}
Jul 20 11:09:30.332664 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.332629    2577 scope.go:117] "RemoveContainer" containerID="76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b"
Jul 20 11:09:30.341507 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.341491    2577 scope.go:117] "RemoveContainer" containerID="76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b"
Jul 20 11:09:30.341869 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:09:30.341783    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b\": container with ID starting with 76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b not found: ID does not exist" containerID="76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b"
Jul 20 11:09:30.341869 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.341810    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b"} err="failed to get container status \"76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b\": rpc error: code = NotFound desc = could not find container \"76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b\": container with ID starting with 76c771d31bdc17f1dfdbae32243077ac087eee2c4619aa5b4ab20c881347203b not found: ID does not exist"
Jul 20 11:09:30.351459 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.351427    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 20 11:09:30.355338 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:30.355311    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 20 11:09:31.796758 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:31.796715    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4deef5d-09d0-4243-9828-b8713db02ff9" path="/var/lib/kubelet/pods/b4deef5d-09d0-4243-9828-b8713db02ff9/volumes"
Jul 20 11:09:39.848281 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.848245    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 20 11:09:39.848714 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.848700    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4deef5d-09d0-4243-9828-b8713db02ff9" containerName="main"
Jul 20 11:09:39.848760 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.848716    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4deef5d-09d0-4243-9828-b8713db02ff9" containerName="main"
Jul 20 11:09:39.848804 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.848795    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4deef5d-09d0-4243-9828-b8713db02ff9" containerName="main"
Jul 20 11:09:39.853583 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.853560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:39.856557 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.856500    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 20 11:09:39.856743 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.856569    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 20 11:09:39.856743 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.856583    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-tznxb\""
Jul 20 11:09:39.856743 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.856592    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 20 11:09:39.856743 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.856583    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-hkrp5\""
Jul 20 11:09:39.864071 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.864042    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 20 11:09:39.964084 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.964050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:39.964084 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.964089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:39.964327 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.964127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6qh\" (UniqueName: \"kubernetes.io/projected/dad514f5-9e27-453c-a015-6b69b63317ec-kube-api-access-kz6qh\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:39.964327 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.964232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:39.964327 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.964267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad514f5-9e27-453c-a015-6b69b63317ec-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:39.964327 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:39.964291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065307 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065307 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065588 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6qh\" (UniqueName: \"kubernetes.io/projected/dad514f5-9e27-453c-a015-6b69b63317ec-kube-api-access-kz6qh\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065588 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065588 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad514f5-9e27-453c-a015-6b69b63317ec-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065588 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065811 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065811 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.065811 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.065783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.067867 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.067846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.068066 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.068048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad514f5-9e27-453c-a015-6b69b63317ec-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.074882 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.074857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6qh\" (UniqueName: \"kubernetes.io/projected/dad514f5-9e27-453c-a015-6b69b63317ec-kube-api-access-kz6qh\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.165771 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.165696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:40.305360 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.305333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 20 11:09:40.308474 ip-10-0-137-140 kubenswrapper[2577]: W0720 11:09:40.308451    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad514f5_9e27_453c_a015_6b69b63317ec.slice/crio-6e4d7bec240c1645edf53d6e45d8cf84f086ea8889611423ee205767d4f4827b WatchSource:0}: Error finding container 6e4d7bec240c1645edf53d6e45d8cf84f086ea8889611423ee205767d4f4827b: Status 404 returned error can't find the container with id 6e4d7bec240c1645edf53d6e45d8cf84f086ea8889611423ee205767d4f4827b
Jul 20 11:09:40.387215 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.387180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dad514f5-9e27-453c-a015-6b69b63317ec","Type":"ContainerStarted","Data":"18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8"}
Jul 20 11:09:40.387375 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.387247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dad514f5-9e27-453c-a015-6b69b63317ec","Type":"ContainerStarted","Data":"6e4d7bec240c1645edf53d6e45d8cf84f086ea8889611423ee205767d4f4827b"}
Jul 20 11:09:40.408731 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:40.408672    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podStartSLOduration=1.408654687 podStartE2EDuration="1.408654687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:09:40.407764154 +0000 UTC m=+4391.040030202" watchObservedRunningTime="2026-07-20 11:09:40.408654687 +0000 UTC m=+4391.040920736"
Jul 20 11:09:41.165893 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:41.165857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:50.166587 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:50.166517    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:50.179186 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:50.179162    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:09:50.448808 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:09:50.448728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:10:11.771909 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:11.771823    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:11.789977 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:11.789934    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:11.809671 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:11.809630    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:11.828957 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:11.828913    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:11.847436 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:11.847388    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:11Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:11.847436 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:11.847428    2577 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 20 11:10:22.119195 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:22.119150    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:22.140750 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:22.140702    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:22.159627 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:22.159590    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:22.177689 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:22.177654    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:22.201199 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:22.201170    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:22Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:22.201199 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:22.201199    2577 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 20 11:10:32.584638 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:32.584596    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:32.598593 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:32.598524    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:32.612708 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:32.612668    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:32.633353 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:32.633323    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:32.646306 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:32.646268    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:32Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:32.646306 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:32.646305    2577 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 20 11:10:42.793227 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:42.793180    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:42.808342 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:42.808300    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:42.821850 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:42.821813    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:42.842289 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:42.842254    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:42.857151 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:42.857119    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T11:10:42Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-137-140.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-32f21fc7-bfe9-438c-880b-7e8aaa8ec2d9.svc:9743/node?timeout=10s\": dial tcp 172.30.110.68:9743: connect: connection refused"
Jul 20 11:10:42.857151 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:10:42.857147    2577 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 20 11:11:11.825792 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:11.825758    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-x7926"]
Jul 20 11:11:11.829513 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:11.829492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:11.831476 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:11.831459    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Jul 20 11:11:12.006905 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.006872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57075235-eeed-4395-aa24-4bc85b63530a-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.007080 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.006995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57075235-eeed-4395-aa24-4bc85b63530a-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.007080 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.007025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/57075235-eeed-4395-aa24-4bc85b63530a-ready\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.007080 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.007058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2bp\" (UniqueName: \"kubernetes.io/projected/57075235-eeed-4395-aa24-4bc85b63530a-kube-api-access-8t2bp\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.108598 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.108484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57075235-eeed-4395-aa24-4bc85b63530a-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.108598 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.108548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/57075235-eeed-4395-aa24-4bc85b63530a-ready\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.108598 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.108594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2bp\" (UniqueName: \"kubernetes.io/projected/57075235-eeed-4395-aa24-4bc85b63530a-kube-api-access-8t2bp\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.108874 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.108617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57075235-eeed-4395-aa24-4bc85b63530a-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.108874 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.108624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57075235-eeed-4395-aa24-4bc85b63530a-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.108874 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.108817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/57075235-eeed-4395-aa24-4bc85b63530a-ready\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.109126 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.109108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57075235-eeed-4395-aa24-4bc85b63530a-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.115753 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.115734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2bp\" (UniqueName: \"kubernetes.io/projected/57075235-eeed-4395-aa24-4bc85b63530a-kube-api-access-8t2bp\") pod \"cni-sysctl-allowlist-ds-x7926\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") " pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.138783 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.138757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.151119 ip-10-0-137-140 kubenswrapper[2577]: W0720 11:11:12.151080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57075235_eeed_4395_aa24_4bc85b63530a.slice/crio-e3f0344a8193c86bfdc34f6994fb403d3d64e7b494cb82320b0d944af49a91e8 WatchSource:0}: Error finding container e3f0344a8193c86bfdc34f6994fb403d3d64e7b494cb82320b0d944af49a91e8: Status 404 returned error can't find the container with id e3f0344a8193c86bfdc34f6994fb403d3d64e7b494cb82320b0d944af49a91e8
Jul 20 11:11:12.851139 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.851105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" event={"ID":"57075235-eeed-4395-aa24-4bc85b63530a","Type":"ContainerStarted","Data":"13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b"}
Jul 20 11:11:12.851618 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.851160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" event={"ID":"57075235-eeed-4395-aa24-4bc85b63530a","Type":"ContainerStarted","Data":"e3f0344a8193c86bfdc34f6994fb403d3d64e7b494cb82320b0d944af49a91e8"}
Jul 20 11:11:12.851618 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.851319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:12.870147 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:12.870091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" podStartSLOduration=1.870074905 podStartE2EDuration="1.870074905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 11:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:11:12.868240514 +0000 UTC m=+4483.500506565" watchObservedRunningTime="2026-07-20 11:11:12.870074905 +0000 UTC m=+4483.502340949"
Jul 20 11:11:13.867214 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:13.867185    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:14.849554 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:14.849498    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-x7926"]
Jul 20 11:11:15.865778 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:15.865737    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" podUID="57075235-eeed-4395-aa24-4bc85b63530a" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" gracePeriod=30
Jul 20 11:11:23.857898 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:23.857848    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.858843 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:23.858815    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.859667 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:23.859640    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:23.859770 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:23.859680    2577 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" podUID="57075235-eeed-4395-aa24-4bc85b63530a" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:30.301697 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:30.301582    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:11:30.314453 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:30.314425    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:11:33.857966 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:33.857923    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.859003 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:33.858969    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.860011 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:33.859965    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:33.860110 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:33.860028    2577 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" podUID="57075235-eeed-4395-aa24-4bc85b63530a" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:43.858063 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:43.858019    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.859004 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:43.858968    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.859903 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:43.859872    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 20 11:11:43.859983 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:43.859917    2577 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" podUID="57075235-eeed-4395-aa24-4bc85b63530a" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 20 11:11:45.952384 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:45.952362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-x7926_57075235-eeed-4395-aa24-4bc85b63530a/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:11:45.954688 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:45.954669    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:46.018586 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.018555    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-x7926_57075235-eeed-4395-aa24-4bc85b63530a/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:11:46.018772 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.018599    2577 generic.go:356] "Generic (PLEG): container finished" podID="57075235-eeed-4395-aa24-4bc85b63530a" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b" exitCode=137
Jul 20 11:11:46.018772 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.018631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" event={"ID":"57075235-eeed-4395-aa24-4bc85b63530a","Type":"ContainerDied","Data":"13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b"}
Jul 20 11:11:46.018772 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.018646    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926"
Jul 20 11:11:46.018772 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.018682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-x7926" event={"ID":"57075235-eeed-4395-aa24-4bc85b63530a","Type":"ContainerDied","Data":"e3f0344a8193c86bfdc34f6994fb403d3d64e7b494cb82320b0d944af49a91e8"}
Jul 20 11:11:46.018772 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.018710    2577 scope.go:117] "RemoveContainer" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b"
Jul 20 11:11:46.027716 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.027688    2577 scope.go:117] "RemoveContainer" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b"
Jul 20 11:11:46.027972 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:11:46.027953    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b\": container with ID starting with 13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b not found: ID does not exist" containerID="13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b"
Jul 20 11:11:46.028014 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.027983    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b"} err="failed to get container status \"13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b\": rpc error: code = NotFound desc = could not find container \"13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b\": container with ID starting with 13d70174825b24de9f3663a135ca66d38302d7642a80170d48a64182e90f0f4b not found: ID does not exist"
Jul 20 11:11:46.118522 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.118492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57075235-eeed-4395-aa24-4bc85b63530a-cni-sysctl-allowlist\") pod \"57075235-eeed-4395-aa24-4bc85b63530a\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") "
Jul 20 11:11:46.118692 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.118595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/57075235-eeed-4395-aa24-4bc85b63530a-ready\") pod \"57075235-eeed-4395-aa24-4bc85b63530a\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") "
Jul 20 11:11:46.118734 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.118692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t2bp\" (UniqueName: \"kubernetes.io/projected/57075235-eeed-4395-aa24-4bc85b63530a-kube-api-access-8t2bp\") pod \"57075235-eeed-4395-aa24-4bc85b63530a\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") "
Jul 20 11:11:46.118734 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.118721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57075235-eeed-4395-aa24-4bc85b63530a-tuning-conf-dir\") pod \"57075235-eeed-4395-aa24-4bc85b63530a\" (UID: \"57075235-eeed-4395-aa24-4bc85b63530a\") "
Jul 20 11:11:46.118874 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.118845    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57075235-eeed-4395-aa24-4bc85b63530a-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "57075235-eeed-4395-aa24-4bc85b63530a" (UID: "57075235-eeed-4395-aa24-4bc85b63530a"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 11:11:46.118940 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.118866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/57075235-eeed-4395-aa24-4bc85b63530a-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "57075235-eeed-4395-aa24-4bc85b63530a" (UID: "57075235-eeed-4395-aa24-4bc85b63530a"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Jul 20 11:11:46.118940 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.118882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57075235-eeed-4395-aa24-4bc85b63530a-ready" (OuterVolumeSpecName: "ready") pod "57075235-eeed-4395-aa24-4bc85b63530a" (UID: "57075235-eeed-4395-aa24-4bc85b63530a"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:11:46.119060 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.119044    2577 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57075235-eeed-4395-aa24-4bc85b63530a-tuning-conf-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:11:46.119113 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.119067    2577 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57075235-eeed-4395-aa24-4bc85b63530a-cni-sysctl-allowlist\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:11:46.119113 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.119082    2577 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/57075235-eeed-4395-aa24-4bc85b63530a-ready\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:11:46.121074 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.121012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57075235-eeed-4395-aa24-4bc85b63530a-kube-api-access-8t2bp" (OuterVolumeSpecName: "kube-api-access-8t2bp") pod "57075235-eeed-4395-aa24-4bc85b63530a" (UID: "57075235-eeed-4395-aa24-4bc85b63530a"). InnerVolumeSpecName "kube-api-access-8t2bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:11:46.220320 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.220284    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t2bp\" (UniqueName: \"kubernetes.io/projected/57075235-eeed-4395-aa24-4bc85b63530a-kube-api-access-8t2bp\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:11:46.338110 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.338077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-x7926"]
Jul 20 11:11:46.339763 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:46.339742    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-x7926"]
Jul 20 11:11:47.796340 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:11:47.796299    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57075235-eeed-4395-aa24-4bc85b63530a" path="/var/lib/kubelet/pods/57075235-eeed-4395-aa24-4bc85b63530a/volumes"
Jul 20 11:16:30.335169 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:16:30.335052    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:16:30.346670 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:16:30.346651    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:21:30.366748 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:21:30.366633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:21:30.378082 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:21:30.378061    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:24:35.710762 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:35.710726    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 20 11:24:35.719684 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:24:35.719647    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 20 11:24:35.719684 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:35.719684 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:35.719684 ip-10-0-137-140 kubenswrapper[2577]:  > containerID="18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8" cmd=["/bin/sleep","15"]
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:35.719697    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" message="" err=<
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]:  >
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:24:35.719711    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="dad514f5-9e27-453c-a015-6b69b63317ec" containerName="main" containerID="cri-o://18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8"
Jul 20 11:24:35.720042 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:35.719745    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="dad514f5-9e27-453c-a015-6b69b63317ec" containerName="main" containerID="cri-o://18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8" gracePeriod=60
Jul 20 11:24:35.980423 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:35.980399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:24:36.077799 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.077763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-dshm\") pod \"dad514f5-9e27-453c-a015-6b69b63317ec\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") "
Jul 20 11:24:36.077979 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.077807    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz6qh\" (UniqueName: \"kubernetes.io/projected/dad514f5-9e27-453c-a015-6b69b63317ec-kube-api-access-kz6qh\") pod \"dad514f5-9e27-453c-a015-6b69b63317ec\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") "
Jul 20 11:24:36.077979 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.077829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad514f5-9e27-453c-a015-6b69b63317ec-tls-certs\") pod \"dad514f5-9e27-453c-a015-6b69b63317ec\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") "
Jul 20 11:24:36.077979 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.077877    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-tmp-dir\") pod \"dad514f5-9e27-453c-a015-6b69b63317ec\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") "
Jul 20 11:24:36.077979 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.077895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-home\") pod \"dad514f5-9e27-453c-a015-6b69b63317ec\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") "
Jul 20 11:24:36.077979 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.077957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-model-cache\") pod \"dad514f5-9e27-453c-a015-6b69b63317ec\" (UID: \"dad514f5-9e27-453c-a015-6b69b63317ec\") "
Jul 20 11:24:36.078260 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.078193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dad514f5-9e27-453c-a015-6b69b63317ec" (UID: "dad514f5-9e27-453c-a015-6b69b63317ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:36.078260 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.078211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-home" (OuterVolumeSpecName: "home") pod "dad514f5-9e27-453c-a015-6b69b63317ec" (UID: "dad514f5-9e27-453c-a015-6b69b63317ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:36.078344 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.078312    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "dad514f5-9e27-453c-a015-6b69b63317ec" (UID: "dad514f5-9e27-453c-a015-6b69b63317ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:36.080213 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.080184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad514f5-9e27-453c-a015-6b69b63317ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dad514f5-9e27-453c-a015-6b69b63317ec" (UID: "dad514f5-9e27-453c-a015-6b69b63317ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 11:24:36.080641 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.080623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad514f5-9e27-453c-a015-6b69b63317ec-kube-api-access-kz6qh" (OuterVolumeSpecName: "kube-api-access-kz6qh") pod "dad514f5-9e27-453c-a015-6b69b63317ec" (UID: "dad514f5-9e27-453c-a015-6b69b63317ec"). InnerVolumeSpecName "kube-api-access-kz6qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 11:24:36.080699 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.080638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-dshm" (OuterVolumeSpecName: "dshm") pod "dad514f5-9e27-453c-a015-6b69b63317ec" (UID: "dad514f5-9e27-453c-a015-6b69b63317ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 11:24:36.179557 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.179505    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-model-cache\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:24:36.179557 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.179558    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-dshm\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:24:36.179769 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.179568    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz6qh\" (UniqueName: \"kubernetes.io/projected/dad514f5-9e27-453c-a015-6b69b63317ec-kube-api-access-kz6qh\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:24:36.179769 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.179579    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad514f5-9e27-453c-a015-6b69b63317ec-tls-certs\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:24:36.179769 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.179587    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-tmp-dir\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:24:36.179769 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.179595    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad514f5-9e27-453c-a015-6b69b63317ec-home\") on node \"ip-10-0-137-140.ec2.internal\" DevicePath \"\""
Jul 20 11:24:36.766076 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.766043    2577 generic.go:356] "Generic (PLEG): container finished" podID="dad514f5-9e27-453c-a015-6b69b63317ec" containerID="18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8" exitCode=0
Jul 20 11:24:36.766571 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.766088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 20 11:24:36.766571 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.766097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dad514f5-9e27-453c-a015-6b69b63317ec","Type":"ContainerDied","Data":"18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8"}
Jul 20 11:24:36.766571 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.766141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dad514f5-9e27-453c-a015-6b69b63317ec","Type":"ContainerDied","Data":"6e4d7bec240c1645edf53d6e45d8cf84f086ea8889611423ee205767d4f4827b"}
Jul 20 11:24:36.766571 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.766162    2577 scope.go:117] "RemoveContainer" containerID="18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8"
Jul 20 11:24:36.776859 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.776836    2577 scope.go:117] "RemoveContainer" containerID="18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8"
Jul 20 11:24:36.777201 ip-10-0-137-140 kubenswrapper[2577]: E0720 11:24:36.777170    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8\": container with ID starting with 18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8 not found: ID does not exist" containerID="18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8"
Jul 20 11:24:36.777314 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.777215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8"} err="failed to get container status \"18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8\": rpc error: code = NotFound desc = could not find container \"18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8\": container with ID starting with 18765d69931d18649d7c3a52641df1ef9893363e3e2ab58debcc0d2b36fa5ef8 not found: ID does not exist"
Jul 20 11:24:36.791063 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.791028    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 20 11:24:36.793889 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:36.793868    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 20 11:24:37.796806 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:24:37.796773    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad514f5-9e27-453c-a015-6b69b63317ec" path="/var/lib/kubelet/pods/dad514f5-9e27-453c-a015-6b69b63317ec/volumes"
Jul 20 11:26:12.607603 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:12.607572    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6bfd579b-gk6x5_cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1/router/0.log"
Jul 20 11:26:13.445827 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:13.445802    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6bfd579b-gk6x5_cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1/router/0.log"
Jul 20 11:26:14.149814 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:14.149785    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-mjgp9_ada649ba-530d-4e2d-bebf-6a247a7f10a8/authorino/0.log"
Jul 20 11:26:14.187851 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:14.187824    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-ld9rx_e6dad072-4b4d-4f90-b192-0ae8733f48be/manager/0.log"
Jul 20 11:26:14.366486 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:14.366453    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-r96fg_011a6bc5-93c1-438b-a52d-ba0b9675f845/manager/0.log"
Jul 20 11:26:20.440909 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.440872    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4rtcj/must-gather-xsljk"]
Jul 20 11:26:20.441427 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.441409    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad514f5-9e27-453c-a015-6b69b63317ec" containerName="main"
Jul 20 11:26:20.441506 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.441431    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad514f5-9e27-453c-a015-6b69b63317ec" containerName="main"
Jul 20 11:26:20.441506 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.441467    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57075235-eeed-4395-aa24-4bc85b63530a" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:26:20.441506 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.441477    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="57075235-eeed-4395-aa24-4bc85b63530a" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:26:20.441721 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.441599    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="57075235-eeed-4395-aa24-4bc85b63530a" containerName="kube-multus-additional-cni-plugins"
Jul 20 11:26:20.441721 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.441618    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad514f5-9e27-453c-a015-6b69b63317ec" containerName="main"
Jul 20 11:26:20.444998 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.444978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.446831 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.446812    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rtcj\"/\"openshift-service-ca.crt\""
Jul 20 11:26:20.447647 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.447605    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4rtcj\"/\"default-dockercfg-dh72t\""
Jul 20 11:26:20.447781 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.447649    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rtcj\"/\"kube-root-ca.crt\""
Jul 20 11:26:20.450352 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.450333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/must-gather-xsljk"]
Jul 20 11:26:20.613864 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.613826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42a5449a-9782-4b0e-8683-2b5ac54a3c03-must-gather-output\") pod \"must-gather-xsljk\" (UID: \"42a5449a-9782-4b0e-8683-2b5ac54a3c03\") " pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.614045 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.613917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pmk\" (UniqueName: \"kubernetes.io/projected/42a5449a-9782-4b0e-8683-2b5ac54a3c03-kube-api-access-89pmk\") pod \"must-gather-xsljk\" (UID: \"42a5449a-9782-4b0e-8683-2b5ac54a3c03\") " pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.714945 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.714922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pmk\" (UniqueName: \"kubernetes.io/projected/42a5449a-9782-4b0e-8683-2b5ac54a3c03-kube-api-access-89pmk\") pod \"must-gather-xsljk\" (UID: \"42a5449a-9782-4b0e-8683-2b5ac54a3c03\") " pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.715103 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.715021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42a5449a-9782-4b0e-8683-2b5ac54a3c03-must-gather-output\") pod \"must-gather-xsljk\" (UID: \"42a5449a-9782-4b0e-8683-2b5ac54a3c03\") " pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.715372 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.715353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/42a5449a-9782-4b0e-8683-2b5ac54a3c03-must-gather-output\") pod \"must-gather-xsljk\" (UID: \"42a5449a-9782-4b0e-8683-2b5ac54a3c03\") " pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.722279 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.722255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pmk\" (UniqueName: \"kubernetes.io/projected/42a5449a-9782-4b0e-8683-2b5ac54a3c03-kube-api-access-89pmk\") pod \"must-gather-xsljk\" (UID: \"42a5449a-9782-4b0e-8683-2b5ac54a3c03\") " pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.754979 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.754951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/must-gather-xsljk"
Jul 20 11:26:20.888084 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.888058    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/must-gather-xsljk"]
Jul 20 11:26:20.890794 ip-10-0-137-140 kubenswrapper[2577]: W0720 11:26:20.890764    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42a5449a_9782_4b0e_8683_2b5ac54a3c03.slice/crio-dd21a674201981747e58ea117678b4196eacdd2873c89df0dda2595bfbb9c763 WatchSource:0}: Error finding container dd21a674201981747e58ea117678b4196eacdd2873c89df0dda2595bfbb9c763: Status 404 returned error can't find the container with id dd21a674201981747e58ea117678b4196eacdd2873c89df0dda2595bfbb9c763
Jul 20 11:26:20.892738 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:20.892722    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 11:26:21.292974 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:21.292932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/must-gather-xsljk" event={"ID":"42a5449a-9782-4b0e-8683-2b5ac54a3c03","Type":"ContainerStarted","Data":"dd21a674201981747e58ea117678b4196eacdd2873c89df0dda2595bfbb9c763"}
Jul 20 11:26:22.304012 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:22.303933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/must-gather-xsljk" event={"ID":"42a5449a-9782-4b0e-8683-2b5ac54a3c03","Type":"ContainerStarted","Data":"152e2227e82e0fcdaf34ff966b5e78aaaec24adbbe4e0fcbf55ed07097bcd79e"}
Jul 20 11:26:22.304012 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:22.303984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/must-gather-xsljk" event={"ID":"42a5449a-9782-4b0e-8683-2b5ac54a3c03","Type":"ContainerStarted","Data":"fbb56b047d3842d6e4deba26e6e43f2bf2b15599d2050c3b9d78e31b06b921e2"}
Jul 20 11:26:22.316780 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:22.316712    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4rtcj/must-gather-xsljk" podStartSLOduration=1.258366074 podStartE2EDuration="2.316691926s" totalImagesPullingTime="1.058325852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 11:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:26:22.316178167 +0000 UTC m=+5392.948444216" watchObservedRunningTime="2026-07-20 11:26:22.316691926 +0000 UTC m=+5392.948957977"
Jul 20 11:26:23.528767 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:23.528736    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tjzjf_acf5741f-e17e-4366-bd46-a3ec4dd047ce/global-pull-secret-syncer/0.log"
Jul 20 11:26:23.626164 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:23.626129    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dgsrm_a2fdfc3b-ca3a-4a6f-873d-20de0e9b934c/konnectivity-agent/0.log"
Jul 20 11:26:23.736234 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:23.736200    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-140.ec2.internal_5b0439c32b7ecbcfae8e78a0f3767df0/haproxy/0.log"
Jul 20 11:26:28.273202 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:28.273169    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-mjgp9_ada649ba-530d-4e2d-bebf-6a247a7f10a8/authorino/0.log"
Jul 20 11:26:28.357242 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:28.357166    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-ld9rx_e6dad072-4b4d-4f90-b192-0ae8733f48be/manager/0.log"
Jul 20 11:26:28.528334 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:28.528296    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-r96fg_011a6bc5-93c1-438b-a52d-ba0b9675f845/manager/0.log"
Jul 20 11:26:30.025024 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.024900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0f469a0-2712-4469-9c75-560c32397361/alertmanager/0.log"
Jul 20 11:26:30.049391 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.049340    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0f469a0-2712-4469-9c75-560c32397361/config-reloader/0.log"
Jul 20 11:26:30.074415 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.074372    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0f469a0-2712-4469-9c75-560c32397361/kube-rbac-proxy-web/0.log"
Jul 20 11:26:30.099818 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.099795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0f469a0-2712-4469-9c75-560c32397361/kube-rbac-proxy/0.log"
Jul 20 11:26:30.124116 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.124090    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0f469a0-2712-4469-9c75-560c32397361/kube-rbac-proxy-metric/0.log"
Jul 20 11:26:30.148071 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.148046    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0f469a0-2712-4469-9c75-560c32397361/prom-label-proxy/0.log"
Jul 20 11:26:30.173523 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.173492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0f469a0-2712-4469-9c75-560c32397361/init-config-reloader/0.log"
Jul 20 11:26:30.251806 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.251775    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jlgj7_731c943c-9af0-4cdd-a209-1c8d40defcf1/kube-state-metrics/0.log"
Jul 20 11:26:30.275104 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.275035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jlgj7_731c943c-9af0-4cdd-a209-1c8d40defcf1/kube-rbac-proxy-main/0.log"
Jul 20 11:26:30.299797 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.299765    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jlgj7_731c943c-9af0-4cdd-a209-1c8d40defcf1/kube-rbac-proxy-self/0.log"
Jul 20 11:26:30.329044 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.329014    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76c658bcb6-46bmb_62209b79-67dd-4b97-ade3-0b3c017141d7/metrics-server/0.log"
Jul 20 11:26:30.358354 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.358313    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-97vtd_b3ce2356-5a10-4a03-afb1-98ce88d4f4fe/monitoring-plugin/0.log"
Jul 20 11:26:30.450993 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.450965    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:26:30.461553 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.460378    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:26:30.492077 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.492044    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nqbpx_b315ab5e-5d34-4c17-be7f-777549b6ce3a/node-exporter/0.log"
Jul 20 11:26:30.515813 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.515782    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nqbpx_b315ab5e-5d34-4c17-be7f-777549b6ce3a/kube-rbac-proxy/0.log"
Jul 20 11:26:30.541479 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.541403    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nqbpx_b315ab5e-5d34-4c17-be7f-777549b6ce3a/init-textfile/0.log"
Jul 20 11:26:30.655881 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.655848    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-24hlw_60023925-65ad-470d-aa92-bfe998e9e857/kube-rbac-proxy-main/0.log"
Jul 20 11:26:30.680895 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.680867    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-24hlw_60023925-65ad-470d-aa92-bfe998e9e857/kube-rbac-proxy-self/0.log"
Jul 20 11:26:30.709031 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.709000    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-24hlw_60023925-65ad-470d-aa92-bfe998e9e857/openshift-state-metrics/0.log"
Jul 20 11:26:30.761265 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.761238    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6bd30472-f2aa-4c90-a501-9440e9492ab6/prometheus/0.log"
Jul 20 11:26:30.779700 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.779544    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6bd30472-f2aa-4c90-a501-9440e9492ab6/config-reloader/0.log"
Jul 20 11:26:30.805776 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.805698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6bd30472-f2aa-4c90-a501-9440e9492ab6/thanos-sidecar/0.log"
Jul 20 11:26:30.831190 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.831154    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6bd30472-f2aa-4c90-a501-9440e9492ab6/kube-rbac-proxy-web/0.log"
Jul 20 11:26:30.856412 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.856385    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6bd30472-f2aa-4c90-a501-9440e9492ab6/kube-rbac-proxy/0.log"
Jul 20 11:26:30.883360 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.883299    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6bd30472-f2aa-4c90-a501-9440e9492ab6/kube-rbac-proxy-thanos/0.log"
Jul 20 11:26:30.910111 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:30.910079    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6bd30472-f2aa-4c90-a501-9440e9492ab6/init-config-reloader/0.log"
Jul 20 11:26:31.161598 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:31.161568    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6597b86945-qg9rk_0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294/thanos-query/0.log"
Jul 20 11:26:31.194667 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:31.194638    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6597b86945-qg9rk_0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294/kube-rbac-proxy-web/0.log"
Jul 20 11:26:31.228074 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:31.228045    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6597b86945-qg9rk_0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294/kube-rbac-proxy/0.log"
Jul 20 11:26:31.258916 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:31.258885    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6597b86945-qg9rk_0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294/prom-label-proxy/0.log"
Jul 20 11:26:31.287267 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:31.287236    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6597b86945-qg9rk_0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294/kube-rbac-proxy-rules/0.log"
Jul 20 11:26:31.320605 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:31.320574    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6597b86945-qg9rk_0ea8561e-a9a8-4e77-99ef-5a6ae6c3e294/kube-rbac-proxy-metrics/0.log"
Jul 20 11:26:33.203905 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:33.203875    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/1.log"
Jul 20 11:26:33.211747 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:33.211721    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-djbdl_dc4c69cf-e0a0-410e-a9df-e7f0f8319ecb/console-operator/2.log"
Jul 20 11:26:33.660710 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:33.660638    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ff5665d54-7bdbv_f368923e-74a5-4de6-8cb9-ed4a1507e37c/console/0.log"
Jul 20 11:26:34.123348 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.123304    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"]
Jul 20 11:26:34.128583 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.128551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.136045 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.136020    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"]
Jul 20 11:26:34.159952 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.159908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k596l\" (UniqueName: \"kubernetes.io/projected/b063c752-07ea-4ae9-8eee-07f1876e6d72-kube-api-access-k596l\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.160256 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.160223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-sys\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.160427 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.160401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-proc\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.160575 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.160559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-lib-modules\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.160719 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.160705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-podres\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.262159 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k596l\" (UniqueName: \"kubernetes.io/projected/b063c752-07ea-4ae9-8eee-07f1876e6d72-kube-api-access-k596l\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.262656 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-sys\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.262656 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-proc\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.262656 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-lib-modules\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.262656 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-podres\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.262656 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-podres\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.262944 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-sys\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.263019 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.262998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-proc\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.263135 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.263114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b063c752-07ea-4ae9-8eee-07f1876e6d72-lib-modules\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.270187 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.270161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k596l\" (UniqueName: \"kubernetes.io/projected/b063c752-07ea-4ae9-8eee-07f1876e6d72-kube-api-access-k596l\") pod \"perf-node-gather-daemonset-mthzt\" (UID: \"b063c752-07ea-4ae9-8eee-07f1876e6d72\") " pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.321186 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.321154    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-dhvkq_7925848d-a13e-43e9-ad35-caeb2dcc5357/volume-data-source-validator/0.log"
Jul 20 11:26:34.443839 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.443752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:34.609467 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:34.609439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"]
Jul 20 11:26:34.612752 ip-10-0-137-140 kubenswrapper[2577]: W0720 11:26:34.612715    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb063c752_07ea_4ae9_8eee_07f1876e6d72.slice/crio-3f4281102b9f9443f5c350706cb9e69f9bd0a9840c473ce39b65c1425196aa7e WatchSource:0}: Error finding container 3f4281102b9f9443f5c350706cb9e69f9bd0a9840c473ce39b65c1425196aa7e: Status 404 returned error can't find the container with id 3f4281102b9f9443f5c350706cb9e69f9bd0a9840c473ce39b65c1425196aa7e
Jul 20 11:26:35.406147 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:35.406110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt" event={"ID":"b063c752-07ea-4ae9-8eee-07f1876e6d72","Type":"ContainerStarted","Data":"3111f629a202833faf49686a1b8a9497f1c77c91502a8320bafd0ae8f51c30b4"}
Jul 20 11:26:35.406647 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:35.406175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt" event={"ID":"b063c752-07ea-4ae9-8eee-07f1876e6d72","Type":"ContainerStarted","Data":"3f4281102b9f9443f5c350706cb9e69f9bd0a9840c473ce39b65c1425196aa7e"}
Jul 20 11:26:35.406647 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:35.406568    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:35.526211 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:35.526182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8bgp6_7d996786-7f9f-493d-a44b-c7010b8f790a/dns/0.log"
Jul 20 11:26:35.549630 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:35.549607    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8bgp6_7d996786-7f9f-493d-a44b-c7010b8f790a/kube-rbac-proxy/0.log"
Jul 20 11:26:35.646075 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:35.646046    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wmz4_29c3298f-7d97-47d3-a41a-b064e8935ff1/dns-node-resolver/0.log"
Jul 20 11:26:36.300499 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:36.300457    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x48x8_e0a99480-abae-43bb-86a7-47ef8e6836d2/node-ca/0.log"
Jul 20 11:26:37.401116 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:37.401089    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6bfd579b-gk6x5_cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1/router/0.log"
Jul 20 11:26:37.965764 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:37.965732    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w6n4p_3f39cb14-5b06-42e5-82b1-fd5918ce8120/serve-healthcheck-canary/0.log"
Jul 20 11:26:38.643157 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:38.643075    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7t54_55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70/kube-rbac-proxy/0.log"
Jul 20 11:26:38.665001 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:38.664973    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7t54_55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70/exporter/0.log"
Jul 20 11:26:38.687216 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:38.687181    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7t54_55a9ae5e-7b1a-47de-b3e9-b036e1cd0e70/extractor/0.log"
Jul 20 11:26:41.424061 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:41.424031    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt"
Jul 20 11:26:41.439861 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:41.439805    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4rtcj/perf-node-gather-daemonset-mthzt" podStartSLOduration=7.43978342 podStartE2EDuration="7.43978342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 11:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 11:26:35.421673813 +0000 UTC m=+5406.053939864" watchObservedRunningTime="2026-07-20 11:26:41.43978342 +0000 UTC m=+5412.072049469"
Jul 20 11:26:41.753883 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:41.753849    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86dd6ccfdd-57rh6_d6841639-44d7-475f-a8d5-39dfdfc38b14/manager/0.log"
Jul 20 11:26:41.822518 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:41.822492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7msm6_dbc23d97-137f-44b8-8cce-5de9186d2309/openshift-lws-operator/0.log"
Jul 20 11:26:42.588906 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:42.588883    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-rv7mc_2fa349bd-c1e5-4d15-a82b-db82003e356e/server/0.log"
Jul 20 11:26:43.193412 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:43.193313    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-d7pgn_75d105d0-7ebe-46cb-abf0-230364e31e60/manager/0.log"
Jul 20 11:26:43.763883 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:43.763857    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6bfd579b-gk6x5_cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1/router/0.log"
Jul 20 11:26:45.013730 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:45.013694    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6bfd579b-gk6x5_cfddf79e-8aa4-4d31-98e5-6c107ff6a1e1/router/0.log"
Jul 20 11:26:48.378036 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:48.378011    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-t5p2l_d4dafae4-d1e1-4750-9482-793dc605e0fd/migrator/0.log"
Jul 20 11:26:48.403152 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:48.403123    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-t5p2l_d4dafae4-d1e1-4750-9482-793dc605e0fd/graceful-termination/0.log"
Jul 20 11:26:49.904826 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:49.904799    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbc85_dec8fc27-3040-4b3f-91ea-b6696b3aae15/kube-multus-additional-cni-plugins/0.log"
Jul 20 11:26:49.928289 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:49.928263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbc85_dec8fc27-3040-4b3f-91ea-b6696b3aae15/egress-router-binary-copy/0.log"
Jul 20 11:26:49.956206 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:49.956178    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbc85_dec8fc27-3040-4b3f-91ea-b6696b3aae15/cni-plugins/0.log"
Jul 20 11:26:49.978851 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:49.978827    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbc85_dec8fc27-3040-4b3f-91ea-b6696b3aae15/bond-cni-plugin/0.log"
Jul 20 11:26:50.001296 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:50.001264    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbc85_dec8fc27-3040-4b3f-91ea-b6696b3aae15/routeoverride-cni/0.log"
Jul 20 11:26:50.025497 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:50.025475    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbc85_dec8fc27-3040-4b3f-91ea-b6696b3aae15/whereabouts-cni-bincopy/0.log"
Jul 20 11:26:50.048682 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:50.048659    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbc85_dec8fc27-3040-4b3f-91ea-b6696b3aae15/whereabouts-cni/0.log"
Jul 20 11:26:50.333044 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:50.333016    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w49lz_bb539c41-358c-4ed9-8ae4-f6b4e43d35f0/kube-multus/0.log"
Jul 20 11:26:50.451574 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:50.451517    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mg256_7da1e17e-2528-47b6-90a5-cc7f4d4103a8/network-metrics-daemon/0.log"
Jul 20 11:26:50.473216 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:50.473187    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mg256_7da1e17e-2528-47b6-90a5-cc7f4d4103a8/kube-rbac-proxy/0.log"
Jul 20 11:26:51.317961 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.317929    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/ovn-controller/0.log"
Jul 20 11:26:51.336878 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.336848    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/ovn-acl-logging/0.log"
Jul 20 11:26:51.364248 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.364222    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/kube-rbac-proxy-node/0.log"
Jul 20 11:26:51.388549 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.388508    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 20 11:26:51.409333 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.409309    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/northd/0.log"
Jul 20 11:26:51.432652 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.432624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/nbdb/0.log"
Jul 20 11:26:51.454183 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.454161    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/sbdb/0.log"
Jul 20 11:26:51.518933 ip-10-0-137-140 kubenswrapper[2577]: I0720 11:26:51.518902    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h6hb_8c74b748-ca84-4956-a79b-c56f113ae50a/ovnkube-controller/0.log"