Jul 08 19:04:14.442827 ip-10-0-140-33 systemd[1]: Starting Kubernetes Kubelet... Jul 08 19:04:14.826138 ip-10-0-140-33 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:04:14.826138 ip-10-0-140-33 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 19:04:14.826138 ip-10-0-140-33 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:04:14.826138 ip-10-0-140-33 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 19:04:14.826138 ip-10-0-140-33 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:04:14.827405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.826126 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 19:04:14.830924 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830902 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:04:14.830924 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830920 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:04:14.830924 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830925 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:04:14.830924 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830929 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:04:14.830924 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830933 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830937 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830941 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830946 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830949 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830953 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830956 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830960 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830963 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830967 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830970 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830974 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830978 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830982 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830985 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830989 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830993 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.830996 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831001 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831004 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:04:14.831216 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831008 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831012 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831016 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831020 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831033 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831037 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831041 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831045 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831049 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831053 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831057 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831061 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831066 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831071 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831076 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831081 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831085 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831089 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831093 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831097 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:04:14.832050 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831102 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831106 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831110 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831114 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831119 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831123 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831127 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831131 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831135 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831140 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831144 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831148 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831153 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831158 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831164 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831168 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831173 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831177 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831182 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831187 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:04:14.832914 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831191 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831197 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831204 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831209 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831213 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831217 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831224 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831229 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831236 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831240 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831245 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831249 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831255 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831261 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831266 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831273 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831277 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831281 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831285 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:04:14.833795 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831289 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831294 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831298 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831302 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831306 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831310 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831315 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831318 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831323 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831327 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831331 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831337 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831342 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831346 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831350 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.831354 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832518 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832528 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832533 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:04:14.834617 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832557 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832562 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832567 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832572 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832576 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832580 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832584 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832588 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832592 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832596 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832601 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832605 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832610 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832614 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832618 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832623 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832626 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832630 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832635 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:04:14.835405 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832639 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832644 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832651 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832656 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832661 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832666 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832671 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832675 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832680 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832684 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832689 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832693 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832697 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832701 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832706 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832709 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832714 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832718 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832723 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832727 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:04:14.836023 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832731 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832735 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832739 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832744 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832749 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832753 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832757 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832761 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832765 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832769 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832773 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832778 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832782 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832786 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832790 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832794 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832800 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832805 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832810 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:04:14.836700 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832814 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832819 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832823 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832827 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832831 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832835 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832839 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832845 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832852 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832856 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832860 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832867 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832871 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832875 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832880 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832884 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832888 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832892 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832897 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832901 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:04:14.837174 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832905 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832910 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832913 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832917 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832921 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832926 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832930 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832934 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832939 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832943 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832948 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832955 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832960 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832965 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832969 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832974 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832978 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.832982 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833579 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833588 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:04:14.837884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833593 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833597 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833601 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833606 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833610 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833615 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833620 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833624 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833628 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833632 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833638 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833643 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833647 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833652 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833656 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833660 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833665 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833669 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833673 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833679 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:04:14.838453 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833685 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833689 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833694 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833699 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833703 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833709 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833713 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833717 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833721 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833725 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833729 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833734 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833738 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833742 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833746 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833750 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833754 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833758 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833762 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833767 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:04:14.838967 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833771 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833775 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833779 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833785 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833789 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833793 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833797 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833801 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833805 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833809 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833813 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833817 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833821 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833825 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833830 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833834 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833839 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833843 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833847 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833851 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:04:14.839523 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833856 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833860 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833864 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833868 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833875 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833881 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833886 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833890 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833895 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833899 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833904 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833908 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833912 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833917 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833921 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833925 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833930 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833936 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833940 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:04:14.840020 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833944 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833948 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833954 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833960 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833964 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833969 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833974 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833978 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833982 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833987 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833991 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833995 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.833999 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.834003 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.834007 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.834012 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.834016 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.834020 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834124 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834135 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 19:04:14.840499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834145 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834151 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834158 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834163 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834170 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834176 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834181 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834186 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834191 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834197 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834202 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834207 2569 flags.go:64] FLAG: --cgroup-root="" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834212 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834216 2569 flags.go:64] FLAG: --client-ca-file="" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834224 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834229 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834235 2569 flags.go:64] FLAG: --cluster-domain="" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834239 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834245 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834250 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834255 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834261 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834266 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 19:04:14.840988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834271 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834276 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834281 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834285 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834290 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834295 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834299 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834309 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834314 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834319 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834324 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834328 2569 flags.go:64] FLAG: --enable-server="true" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834333 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834341 2569 flags.go:64] FLAG: --event-burst="100" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834345 2569 flags.go:64] FLAG: --event-qps="50" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834350 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834355 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834359 2569 flags.go:64] FLAG: --eviction-hard="" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834366 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834371 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834377 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834382 2569 flags.go:64] FLAG: --eviction-soft="" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834387 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834394 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834398 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 19:04:14.841531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834403 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834407 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834412 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834417 2569 flags.go:64] FLAG: --feature-gates="" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834423 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834427 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834432 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834437 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834442 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834447 2569 flags.go:64] FLAG: --help="false" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834452 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-33.ec2.internal" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834457 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834461 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834466 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834471 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834477 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834481 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834486 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834490 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834495 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834499 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834504 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834509 2569 flags.go:64] FLAG: --kube-reserved="" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834514 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 19:04:14.842162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834519 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834524 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834530 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834554 2569 flags.go:64] FLAG: --lock-file="" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834561 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834566 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834572 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834581 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834586 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834591 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834595 2569 flags.go:64] FLAG: --logging-format="text" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834600 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834605 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834610 2569 flags.go:64] FLAG: --manifest-url="" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834614 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834620 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834625 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834631 2569 flags.go:64] FLAG: --max-pods="110" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834635 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834640 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834644 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834649 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834654 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834659 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834664 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 19:04:14.842886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834676 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834680 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834685 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834689 2569 flags.go:64] FLAG: --pod-cidr="" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834694 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834702 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834707 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834712 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834716 2569 flags.go:64] FLAG: --port="10250" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834724 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834728 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-028499413a4af6df4" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834733 2569 flags.go:64] FLAG: --qos-reserved="" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834739 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834744 2569 flags.go:64] FLAG: --register-node="true" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834750 2569 flags.go:64] FLAG: --register-with-taints="" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834756 2569 flags.go:64] FLAG: --registry-burst="10" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834761 2569 flags.go:64] FLAG: --registry-qps="5" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834766 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834770 2569 flags.go:64] FLAG: --reserved-memory="" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834776 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834781 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834786 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834791 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834795 2569 flags.go:64] FLAG: --runonce="false" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834800 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 19:04:14.843494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834805 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834809 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834814 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834818 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834823 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834828 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834833 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834837 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834842 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834846 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834850 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834855 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834860 2569 flags.go:64] FLAG: --system-cgroups="" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834864 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834873 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834877 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834883 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834889 2569 flags.go:64] FLAG: --tls-min-version="" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834893 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834898 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834903 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834910 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834915 2569 flags.go:64] FLAG: --v="2" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834921 2569 flags.go:64] FLAG: --version="false" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834927 2569 flags.go:64] FLAG: --vmodule="" Jul 08 19:04:14.844133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834932 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.834937 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835156 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835163 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835169 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835174 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835179 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835183 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835188 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835192 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835197 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835200 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835205 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835209 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835213 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835217 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835221 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835225 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835229 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835233 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:04:14.844772 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835237 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835241 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835245 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835252 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835259 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835265 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835270 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835275 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835289 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835295 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835300 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835304 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835309 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835314 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835318 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835323 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835327 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835331 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:04:14.845254 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835336 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835340 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835344 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835349 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835353 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835359 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835365 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835372 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835377 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835381 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835385 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835389 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835393 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835397 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835401 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835405 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835409 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835414 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835418 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:04:14.845775 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835422 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835426 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835430 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835437 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835442 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835447 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835451 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835455 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835460 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835464 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835468 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835472 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835476 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835480 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835484 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835488 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835492 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835496 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835500 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835505 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:04:14.846275 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835509 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835513 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835517 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835521 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835525 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835529 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835533 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835555 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835560 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835564 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835570 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835574 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835578 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835582 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835586 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835592 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835596 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835601 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835606 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835611 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:04:14.846789 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835615 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835619 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835623 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.835628 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.836481 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.844066 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.844080 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.844099 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.844108 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844167 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844172 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844175 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844179 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844182 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844185 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:04:14.847268 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844187 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844190 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844193 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844196 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844199 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844202 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844205 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844207 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844210 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844213 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844215 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844218 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844220 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844223 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844225 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844228 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844231 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844234 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844237 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844239 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:04:14.847665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844250 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844253 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844256 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844260 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844264 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844269 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844272 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844275 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844278 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844281 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844283 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844286 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844288 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844291 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844293 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844296 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844298 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844301 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844304 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844306 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:04:14.848146 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844309 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844311 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844314 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844316 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844319 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844321 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844324 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844326 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844329 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844331 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844334 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844337 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844339 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844341 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844344 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844346 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844350 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844352 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844355 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:04:14.848666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844357 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844359 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844362 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844365 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844367 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844370 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844372 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844374 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844377 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844379 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844382 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844385 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844387 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844389 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844392 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844394 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844397 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844400 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844402 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844406 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:04:14.849129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844409 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844412 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844415 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844418 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844420 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844422 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844425 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844427 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844430 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844433 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844439 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844442 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844446 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844449 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:04:14.849665 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.844454 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844583 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844589 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844592 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844595 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844597 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844601 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844604 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844607 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844609 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844612 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844615 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844618 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844621 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844624 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844627 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844629 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844632 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844635 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844637 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:04:14.850012 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844640 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844643 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844645 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844648 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844652 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844656 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844659 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844662 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844664 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844667 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844670 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844673 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844675 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844678 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844680 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844682 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844685 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844687 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844690 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844692 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:04:14.850497 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844695 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844697 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844700 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844702 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844704 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844707 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844709 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844712 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844714 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844717 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844719 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844722 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844725 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844729 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844732 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844735 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844738 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844741 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844743 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:04:14.850982 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844746 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844750 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844753 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844756 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844758 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844761 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844763 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844766 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844768 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844771 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844773 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844776 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844778 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844781 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844784 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844786 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844789 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844791 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844794 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:04:14.851426 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844796 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844798 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844801 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844803 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844806 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844808 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844811 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844813 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844816 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844818 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844821 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844823 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844826 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844828 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844831 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844834 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844836 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844839 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844841 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844844 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:04:14.851904 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844846 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:14.844849 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.844854 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.844967 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.847284 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.848749 2569 server.go:1019] "Starting client certificate rotation" Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.848851 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.849597 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:04:14.852388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.851916 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:04:14.854724 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.854711 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:04:14.864922 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.864898 2569 log.go:25] "Validated CRI v1 runtime API" Jul 08 19:04:14.868034 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.868020 2569 log.go:25] "Validated CRI v1 image API" Jul 08 19:04:14.868793 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.868778 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 19:04:14.871118 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.871101 2569 fs.go:135] Filesystem UUIDs: map[233d7d8d-acab-4912-bab6-9c988fe9dde5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 93e490e5-0b82-48ef-81d0-cbac6dea920d:/dev/nvme0n1p4] Jul 08 19:04:14.871168 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.871119 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 19:04:14.877358 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.877246 2569 manager.go:216] Machine: {Timestamp:2026-07-08 19:04:14.875210813 +0000 UTC m=+0.314513456 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101078 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c740a8d936c0505d22c4e586e8796 SystemUUID:ec2c740a-8d93-6c05-05d2-2c4e586e8796 BootID:b93661a6-5e11-4396-9264-98312a9ab290 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:36:6d:c1:bc:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:36:6d:c1:bc:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:55:04:02:ea:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 19:04:14.877358 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.877356 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 19:04:14.877464 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.877432 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 19:04:14.880716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.880686 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 19:04:14.880859 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.880717 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-33.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 19:04:14.881242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.881227 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:04:14.881674 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.881662 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 19:04:14.881712 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.881676 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 19:04:14.881712 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.881683 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:04:14.882972 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.882959 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:04:14.884645 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.884635 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 19:04:14.888393 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888381 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:04:14.888556 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888533 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 19:04:14.888587 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888577 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 08 19:04:14.888615 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888594 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 19:04:14.888615 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888609 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 19:04:14.888676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888618 2569 kubelet.go:398] "Adding apiserver pod source" Jul 08 19:04:14.888676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888626 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 19:04:14.889003 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.888989 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:04:14.889071 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.889012 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:04:14.890990 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.890966 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 19:04:14.893414 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.893401 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 19:04:14.893464 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.893432 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 19:04:14.894796 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894782 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 19:04:14.894846 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894805 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 19:04:14.894846 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894816 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 19:04:14.894846 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894825 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 19:04:14.894846 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894836 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 19:04:14.894846 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894845 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 19:04:14.894973 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894856 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 19:04:14.894973 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894865 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 19:04:14.894973 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894875 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 19:04:14.894973 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894880 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 19:04:14.894973 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894888 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 19:04:14.894973 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.894893 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 19:04:14.895764 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.895754 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 19:04:14.895764 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.895763 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 19:04:14.898948 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.898930 2569 server.go:1268] "Started kubelet" Jul 08 19:04:14.899037 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.898992 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 19:04:14.899161 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.899056 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 19:04:14.899262 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.899185 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 19:04:14.899829 ip-10-0-140-33 systemd[1]: Started Kubernetes Kubelet. Jul 08 19:04:14.904388 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.904361 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-33.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 19:04:14.904468 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.900639 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 19:04:14.905403 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.905367 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 19:04:14.905527 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.905504 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 19:04:14.905814 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.905725 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 08 19:04:14.905947 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.905923 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 19:04:14.911102 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.910122 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-33.ec2.internal.18c065dd9ada321c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-33.ec2.internal,UID:ip-10-0-140-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-33.ec2.internal,},FirstTimestamp:2026-07-08 19:04:14.898885148 +0000 UTC m=+0.338187770,LastTimestamp:2026-07-08 19:04:14.898885148 +0000 UTC m=+0.338187770,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-33.ec2.internal,}" Jul 08 19:04:14.911505 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.911492 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 19:04:14.911614 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.911593 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:14.911672 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.911607 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 19:04:14.912279 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912262 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 19:04:14.912372 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912296 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 19:04:14.912372 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912312 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 19:04:14.912458 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912404 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 08 19:04:14.912458 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912413 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 08 19:04:14.912573 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912494 2569 factory.go:153] Registering CRI-O factory Jul 08 19:04:14.912613 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912587 2569 factory.go:223] Registration of the crio container factory successfully Jul 08 19:04:14.912657 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912636 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 19:04:14.912657 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912646 2569 factory.go:55] Registering systemd factory Jul 08 19:04:14.912657 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912653 2569 factory.go:223] Registration of the systemd container factory successfully Jul 08 19:04:14.912760 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912672 2569 factory.go:103] Registering Raw factory Jul 08 19:04:14.912760 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.912686 2569 manager.go:1196] Started watching for new ooms in manager Jul 08 19:04:14.913390 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.913364 2569 manager.go:349] Starting recovery of all containers Jul 08 19:04:14.923359 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.923330 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 19:04:14.923481 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.923321 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 19:04:14.923916 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.923888 2569 manager.go:354] Recovery completed Jul 08 19:04:14.927837 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.927826 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:04:14.932305 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.932278 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:04:14.932379 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.932315 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:04:14.932379 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.932325 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:04:14.932815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.932802 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 19:04:14.932815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.932814 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 19:04:14.932897 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.932828 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:04:14.932897 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.932869 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:04:14.933529 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.933515 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:04:14.933593 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.933575 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:04:14.933593 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.933588 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:04:14.934030 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.934018 2569 policy_none.go:49] "None policy: Start" Jul 08 19:04:14.934063 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.934034 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 19:04:14.934063 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.934044 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 19:04:14.934470 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.934408 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-33.ec2.internal.18c065dd9cd8149c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-33.ec2.internal,UID:ip-10-0-140-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-33.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-33.ec2.internal,},FirstTimestamp:2026-07-08 19:04:14.932300956 +0000 UTC m=+0.371603581,LastTimestamp:2026-07-08 19:04:14.932300956 +0000 UTC m=+0.371603581,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-33.ec2.internal,}" Jul 08 19:04:14.935178 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.935164 2569 policy_none.go:47] "Start" Jul 08 19:04:14.956333 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.956255 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-33.ec2.internal.18c065dd9cd85cd0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-33.ec2.internal,UID:ip-10-0-140-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-33.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-33.ec2.internal,},FirstTimestamp:2026-07-08 19:04:14.93231944 +0000 UTC m=+0.371622062,LastTimestamp:2026-07-08 19:04:14.93231944 +0000 UTC m=+0.371622062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-33.ec2.internal,}" Jul 08 19:04:14.961834 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.961809 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vc56b" Jul 08 19:04:14.967731 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.967668 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-33.ec2.internal.18c065dd9cd881cc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-33.ec2.internal,UID:ip-10-0-140-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-33.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-33.ec2.internal,},FirstTimestamp:2026-07-08 19:04:14.932328908 +0000 UTC m=+0.371631530,LastTimestamp:2026-07-08 19:04:14.932328908 +0000 UTC m=+0.371631530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-33.ec2.internal,}" Jul 08 19:04:14.976092 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.976077 2569 manager.go:337] "Starting Device Plugin manager" Jul 08 19:04:14.976193 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.976123 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 19:04:14.976193 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.976136 2569 server.go:87] "Starting device plugin registration server" Jul 08 19:04:14.976386 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.976373 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 19:04:14.976439 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.976389 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 19:04:14.976489 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.976466 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 19:04:14.976557 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.976523 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 19:04:14.976557 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.976529 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 19:04:14.977036 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.977019 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 19:04:14.977105 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.977060 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:14.977992 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:14.977975 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vc56b" Jul 08 19:04:14.981075 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:14.981014 2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-33.ec2.internal.18c065dd9cd8149c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-33.ec2.internal.18c065dd9cd8149c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-33.ec2.internal,UID:ip-10-0-140-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-33.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-33.ec2.internal,},FirstTimestamp:2026-07-08 19:04:14.932300956 +0000 UTC m=+0.371603581,LastTimestamp:2026-07-08 19:04:14.933528568 +0000 UTC m=+0.372831190,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-33.ec2.internal,}" Jul 08 19:04:15.077126 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.077064 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:04:15.078281 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.077964 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:04:15.078281 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.077992 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:04:15.078281 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.078001 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:04:15.078281 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.078030 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.090240 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.090220 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.090315 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.090242 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-33.ec2.internal\": node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.091391 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.091373 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 19:04:15.092641 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.092622 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 19:04:15.092715 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.092656 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 19:04:15.092715 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.092675 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 19:04:15.092715 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.092704 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 19:04:15.095444 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.095428 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:04:15.135924 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.135907 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.193027 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.192998 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal"] Jul 08 19:04:15.193122 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.193063 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:04:15.194131 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.194116 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:04:15.194200 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.194144 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:04:15.194200 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.194154 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:04:15.195462 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.195450 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:04:15.195593 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.195576 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.195632 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.195611 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:04:15.196318 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.196305 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:04:15.196390 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.196311 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:04:15.196390 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.196330 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:04:15.196390 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.196345 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:04:15.196390 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.196346 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:04:15.196518 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.196401 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:04:15.197569 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.197553 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.197623 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.197581 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:04:15.198219 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.198201 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:04:15.198322 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.198232 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:04:15.198322 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.198246 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:04:15.214567 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.214529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/802c451955599a360850d92f832cfcf3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-33.ec2.internal\" (UID: \"802c451955599a360850d92f832cfcf3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.214670 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.214582 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da4443c5c50862dbc167c8fce07acc65-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal\" (UID: \"da4443c5c50862dbc167c8fce07acc65\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.214670 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.214609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4443c5c50862dbc167c8fce07acc65-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal\" (UID: \"da4443c5c50862dbc167c8fce07acc65\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.226322 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.226303 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-33.ec2.internal\" not found" node="ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.230786 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.230771 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-33.ec2.internal\" not found" node="ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.236828 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.236812 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.314716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.314681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da4443c5c50862dbc167c8fce07acc65-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal\" (UID: \"da4443c5c50862dbc167c8fce07acc65\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.314716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.314716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4443c5c50862dbc167c8fce07acc65-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal\" (UID: \"da4443c5c50862dbc167c8fce07acc65\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.314924 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.314742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/802c451955599a360850d92f832cfcf3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-33.ec2.internal\" (UID: \"802c451955599a360850d92f832cfcf3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.314924 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.314787 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/802c451955599a360850d92f832cfcf3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-33.ec2.internal\" (UID: \"802c451955599a360850d92f832cfcf3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.314924 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.314790 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4443c5c50862dbc167c8fce07acc65-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal\" (UID: \"da4443c5c50862dbc167c8fce07acc65\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.314924 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.314787 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da4443c5c50862dbc167c8fce07acc65-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal\" (UID: \"da4443c5c50862dbc167c8fce07acc65\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.337043 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.336990 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.437092 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.437052 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.529849 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.529799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.534865 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.534606 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:15.537931 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.537915 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.639351 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.639263 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.740263 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.740229 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.841363 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.841313 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.848555 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.848511 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 19:04:15.848730 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.848703 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:04:15.942104 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:15.942071 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-33.ec2.internal\" not found" Jul 08 19:04:15.983662 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.983613 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 18:59:15 +0000 UTC" deadline="2028-01-13 00:02:56.732097386 +0000 UTC" Jul 08 19:04:15.983662 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:15.983654 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13276h58m40.748446784s" Jul 08 19:04:16.012705 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.012672 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:04:16.112716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.112694 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" Jul 08 19:04:16.127553 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.127524 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:04:16.129445 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.129427 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" Jul 08 19:04:16.139528 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.139500 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:04:16.216665 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:16.216633 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802c451955599a360850d92f832cfcf3.slice/crio-497675e381b50764e3115ba9d798e7ba6e3463175f7cbd3d9111285ba33eef06 WatchSource:0}: Error finding container 497675e381b50764e3115ba9d798e7ba6e3463175f7cbd3d9111285ba33eef06: Status 404 returned error can't find the container with id 497675e381b50764e3115ba9d798e7ba6e3463175f7cbd3d9111285ba33eef06 Jul 08 19:04:16.221719 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.221699 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:04:16.309534 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.309501 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:04:16.350929 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:16.350900 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4443c5c50862dbc167c8fce07acc65.slice/crio-bdbb8ee361c3df710bffceda4aa4d81985390b023cfd032f70b9ecec00ba7382 WatchSource:0}: Error finding container bdbb8ee361c3df710bffceda4aa4d81985390b023cfd032f70b9ecec00ba7382: Status 404 returned error can't find the container with id bdbb8ee361c3df710bffceda4aa4d81985390b023cfd032f70b9ecec00ba7382 Jul 08 19:04:16.494310 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.494220 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:04:16.744230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.744201 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:04:16.890722 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.890630 2569 apiserver.go:52] "Watching apiserver" Jul 08 19:04:16.898960 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.898929 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 19:04:16.901032 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.901002 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-cdtvk","openshift-ovn-kubernetes/ovnkube-node-jbj8q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb","openshift-cluster-node-tuning-operator/tuned-zmxx5","openshift-image-registry/node-ca-7rlf8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal","openshift-multus/multus-4jqnv","openshift-multus/multus-additional-cni-plugins-vqsjw","openshift-network-diagnostics/network-check-target-t9zpq","openshift-network-operator/iptables-alerter-xdmmm","kube-system/global-pull-secret-syncer-wc6c2","kube-system/konnectivity-agent-p5vml","kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal","openshift-dns/node-resolver-l6jvn"] Jul 08 19:04:16.902480 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.902461 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.903621 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.903599 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:16.903742 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:16.903669 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:16.905284 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.905264 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 19:04:16.905284 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.905276 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 19:04:16.905441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.905282 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 19:04:16.905441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.905277 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 19:04:16.905441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.905318 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2nhd7\"" Jul 08 19:04:16.905666 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.905653 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.907124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.907105 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.908038 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.908017 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 19:04:16.908140 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.908037 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 19:04:16.908140 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.908017 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 19:04:16.908140 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.908099 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mvht8\"" Jul 08 19:04:16.908368 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.908350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:16.909796 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.909667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:16.910700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.910685 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z6zlx\"" Jul 08 19:04:16.910993 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.910979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:16.911057 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:16.911044 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:16.911365 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.911340 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:04:16.911462 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.911343 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 19:04:16.911766 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.911752 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 19:04:16.912028 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.912016 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 19:04:16.912071 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.912039 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 19:04:16.912148 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.912131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 19:04:16.912315 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.912294 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pvkzb\"" Jul 08 19:04:16.912421 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.912342 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 19:04:16.912493 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.912443 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 19:04:16.912744 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.912723 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.913045 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.913027 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kxdnd\"" Jul 08 19:04:16.914170 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.914154 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:16.915147 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.915131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 19:04:16.915227 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.915186 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jpmpp\"" Jul 08 19:04:16.915449 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.915436 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 19:04:16.915586 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.915571 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:16.917269 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.917253 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:16.917381 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:16.917311 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:16.918191 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.918171 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 19:04:16.918463 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.918445 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 19:04:16.918571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.918464 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 19:04:16.918726 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.918710 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 19:04:16.918801 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.918734 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 19:04:16.918801 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.918758 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:16.920323 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.920235 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pdxs6\"" Jul 08 19:04:16.920594 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.920546 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 19:04:16.920719 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.920644 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:04:16.920817 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.920794 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 19:04:16.920993 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.920971 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 19:04:16.921116 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.921098 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75qjl\"" Jul 08 19:04:16.922514 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.922493 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crn2m\"" Jul 08 19:04:16.924137 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.923594 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 19:04:16.924137 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.923879 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 19:04:16.927139 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927118 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:04:16.927397 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-registration-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.927500 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927412 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.927500 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-modprobe-d\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.927500 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-sys\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.927682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927511 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-system-cni-dir\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.927682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927546 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-socket-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.927682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927606 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-device-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.927682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927640 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-lib-modules\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.927682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927667 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-conf-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-hosts-file\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927744 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-cni-bin\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-systemd\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c108ae49-2a6c-44b5-a725-5d3a3b373713-host\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-system-cni-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-os-release\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.927875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-socket-dir-parent\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-hostroot\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-multus-certs\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw27w\" (UniqueName: \"kubernetes.io/projected/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-kube-api-access-lw27w\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c108ae49-2a6c-44b5-a725-5d3a3b373713-serviceca\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.927994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6qn\" (UniqueName: \"kubernetes.io/projected/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-kube-api-access-hh6qn\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-daemon-config\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928048 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmr72\" (UniqueName: \"kubernetes.io/projected/32daff71-c045-4f52-84d2-b4f14425dd63-kube-api-access-jmr72\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-k8s-cni-cncf-io\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-cni-multus\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928160 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:16.928187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-kubernetes\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-tmp-dir\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928262 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-cni-binary-copy\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928288 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysctl-d\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-run\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f794ed16-0d1b-4137-82df-17f6a3830473-etc-tuned\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-cni-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-cnibin\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-etc-kubernetes\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928489 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysctl-conf\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928515 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f794ed16-0d1b-4137-82df-17f6a3830473-tmp\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928541 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld89s\" (UniqueName: \"kubernetes.io/projected/c108ae49-2a6c-44b5-a725-5d3a3b373713-kube-api-access-ld89s\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928591 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-cni-binary-copy\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.928611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-dbus\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-var-lib-kubelet\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928670 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-kubelet\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-sys-fs\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928743 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxf5s\" (UniqueName: \"kubernetes.io/projected/f794ed16-0d1b-4137-82df-17f6a3830473-kube-api-access-kxf5s\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-os-release\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-kubelet-config\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzcvk\" (UniqueName: \"kubernetes.io/projected/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-kube-api-access-vzcvk\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysconfig\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928877 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-host\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-netns\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:16.929145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.928906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-cnibin\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:16.956680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.956661 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7cj29" Jul 08 19:04:16.975257 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:16.975234 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7cj29" Jul 08 19:04:17.013506 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.013480 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 19:04:17.030038 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-cni-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-cnibin\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-etc-kubernetes\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030107 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-cni-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030115 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9423c503-bd8a-4147-9ec9-02137d9fc206-iptables-alerter-script\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.030176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysctl-conf\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.030176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-etc-kubernetes\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f794ed16-0d1b-4137-82df-17f6a3830473-tmp\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-cnibin\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld89s\" (UniqueName: \"kubernetes.io/projected/c108ae49-2a6c-44b5-a725-5d3a3b373713-kube-api-access-ld89s\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030269 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-cni-binary-copy\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-var-lib-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030310 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysctl-conf\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030326 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9423c503-bd8a-4147-9ec9-02137d9fc206-host-slash\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-dbus\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-var-lib-kubelet\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-kubelet\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-kubelet\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.030507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-sys-fs\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxf5s\" (UniqueName: \"kubernetes.io/projected/f794ed16-0d1b-4137-82df-17f6a3830473-kube-api-access-kxf5s\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-os-release\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030587 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-dbus\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030645 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-kubelet-config\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzcvk\" (UniqueName: \"kubernetes.io/projected/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-kube-api-access-vzcvk\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030631 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030927 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-cni-binary-copy\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.030966 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-kubelet\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-var-lib-kubelet\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031042 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-sys-fs\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysconfig\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031079 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-kubelet-config\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031065 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-os-release\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031128 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysconfig\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-host\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-netns\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031193 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-host\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031231 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-netns\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031264 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-cnibin\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-etc-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovn-node-metrics-cert\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-registration-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031330 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-cnibin\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031382 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-modprobe-d\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/428cf648-de10-4fc7-a55a-ffa7aad31575-konnectivity-ca\") pod \"konnectivity-agent-p5vml\" (UID: \"428cf648-de10-4fc7-a55a-ffa7aad31575\") " pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031412 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-registration-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-log-socket\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031481 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-sys\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031493 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-modprobe-d\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031499 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.031953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-run-netns\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031537 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-ovn\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031541 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-sys\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031591 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031662 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk99f\" (UniqueName: \"kubernetes.io/projected/0d613ce2-7191-434e-8b6d-c8960eb95403-kube-api-access-gk99f\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031689 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-system-cni-dir\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-socket-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031740 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-device-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-lib-modules\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031775 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-system-cni-dir\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031877 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-socket-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031884 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-lib-modules\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-conf-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031932 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-hosts-file\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031948 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031955 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-device-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.032576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031966 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-cni-bin\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031973 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-conf-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.031996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrzp\" (UniqueName: \"kubernetes.io/projected/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-kube-api-access-wwrzp\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032008 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-hosts-file\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-cni-bin\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032028 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-node-log\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-cni-netd\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-env-overrides\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-systemd\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032098 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32daff71-c045-4f52-84d2-b4f14425dd63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c108ae49-2a6c-44b5-a725-5d3a3b373713-host\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032137 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c108ae49-2a6c-44b5-a725-5d3a3b373713-host\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-system-cni-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032158 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-systemd\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-os-release\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-socket-dir-parent\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032206 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-system-cni-dir\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032236 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-os-release\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-hostroot\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032267 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-socket-dir-parent\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032281 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-hostroot\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-multus-certs\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw27w\" (UniqueName: \"kubernetes.io/projected/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-kube-api-access-lw27w\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c108ae49-2a6c-44b5-a725-5d3a3b373713-serviceca\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6qn\" (UniqueName: \"kubernetes.io/projected/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-kube-api-access-hh6qn\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032371 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/428cf648-de10-4fc7-a55a-ffa7aad31575-agent-certs\") pod \"konnectivity-agent-p5vml\" (UID: \"428cf648-de10-4fc7-a55a-ffa7aad31575\") " pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovnkube-config\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-daemon-config\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032445 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032496 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmr72\" (UniqueName: \"kubernetes.io/projected/32daff71-c045-4f52-84d2-b4f14425dd63-kube-api-access-jmr72\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032522 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjclj\" (UniqueName: \"kubernetes.io/projected/9423c503-bd8a-4147-9ec9-02137d9fc206-kube-api-access-xjclj\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-k8s-cni-cncf-io\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-cni-multus\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-systemd-units\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032608 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-multus-certs\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.033940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032697 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-run-k8s-cni-cncf-io\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032766 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-host-var-lib-cni-multus\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032878 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c108ae49-2a6c-44b5-a725-5d3a3b373713-serviceca\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovnkube-script-lib\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.032993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-kubernetes\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-tmp-dir\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-cni-binary-copy\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-kubernetes\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.033061 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033076 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-slash\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033118 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-multus-daemon-config\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.033162 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret podName:b1ce6677-e64a-4c21-9faf-a715c7a83d5f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:17.533118254 +0000 UTC m=+2.972420883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret") pod "global-pull-secret-syncer-wc6c2" (UID: "b1ce6677-e64a-4c21-9faf-a715c7a83d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-systemd\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.034853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-cni-bin\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033237 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033295 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033237 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysctl-d\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-run\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033364 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-tmp-dir\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f794ed16-0d1b-4137-82df-17f6a3830473-etc-tuned\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033527 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-etc-sysctl-d\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f794ed16-0d1b-4137-82df-17f6a3830473-run\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033658 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-cni-binary-copy\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.033702 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32daff71-c045-4f52-84d2-b4f14425dd63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.035680 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.034449 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f794ed16-0d1b-4137-82df-17f6a3830473-tmp\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.036259 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.035877 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f794ed16-0d1b-4137-82df-17f6a3830473-etc-tuned\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.039301 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.039279 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:17.039400 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.039307 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:17.039400 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.039321 2569 projected.go:196] Error preparing data for projected volume kube-api-access-znpkq for pod openshift-network-diagnostics/network-check-target-t9zpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:17.039400 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.039381 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq podName:4025f320-f92c-42a0-aa12-69e475af26b9 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:17.539363786 +0000 UTC m=+2.978666420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-znpkq" (UniqueName: "kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq") pod "network-check-target-t9zpq" (UID: "4025f320-f92c-42a0-aa12-69e475af26b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:17.047744 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.047726 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzcvk\" (UniqueName: \"kubernetes.io/projected/4df44483-ba6b-41a4-8bfb-e9c7a3f27436-kube-api-access-vzcvk\") pod \"aws-ebs-csi-driver-node-4nshb\" (UID: \"4df44483-ba6b-41a4-8bfb-e9c7a3f27436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.048403 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.048381 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld89s\" (UniqueName: \"kubernetes.io/projected/c108ae49-2a6c-44b5-a725-5d3a3b373713-kube-api-access-ld89s\") pod \"node-ca-7rlf8\" (UID: \"c108ae49-2a6c-44b5-a725-5d3a3b373713\") " pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:17.049285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.049264 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6qn\" (UniqueName: \"kubernetes.io/projected/e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858-kube-api-access-hh6qn\") pod \"node-resolver-l6jvn\" (UID: \"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858\") " pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:17.052171 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.052150 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw27w\" (UniqueName: \"kubernetes.io/projected/a30ca408-d43a-4fc1-bb33-92de7d6fa99d-kube-api-access-lw27w\") pod \"multus-4jqnv\" (UID: \"a30ca408-d43a-4fc1-bb33-92de7d6fa99d\") " pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.060971 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.060949 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxf5s\" (UniqueName: \"kubernetes.io/projected/f794ed16-0d1b-4137-82df-17f6a3830473-kube-api-access-kxf5s\") pod \"tuned-zmxx5\" (UID: \"f794ed16-0d1b-4137-82df-17f6a3830473\") " pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.063824 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.063762 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmr72\" (UniqueName: \"kubernetes.io/projected/32daff71-c045-4f52-84d2-b4f14425dd63-kube-api-access-jmr72\") pod \"multus-additional-cni-plugins-vqsjw\" (UID: \"32daff71-c045-4f52-84d2-b4f14425dd63\") " pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.101744 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.101641 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" event={"ID":"da4443c5c50862dbc167c8fce07acc65","Type":"ContainerStarted","Data":"bdbb8ee361c3df710bffceda4aa4d81985390b023cfd032f70b9ecec00ba7382"} Jul 08 19:04:17.103693 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.103637 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" event={"ID":"802c451955599a360850d92f832cfcf3","Type":"ContainerStarted","Data":"497675e381b50764e3115ba9d798e7ba6e3463175f7cbd3d9111285ba33eef06"} Jul 08 19:04:17.134748 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-etc-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134748 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovn-node-metrics-cert\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/428cf648-de10-4fc7-a55a-ffa7aad31575-konnectivity-ca\") pod \"konnectivity-agent-p5vml\" (UID: \"428cf648-de10-4fc7-a55a-ffa7aad31575\") " pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-log-socket\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134821 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-run-netns\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-ovn\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134869 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134905 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-log-socket\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk99f\" (UniqueName: \"kubernetes.io/projected/0d613ce2-7191-434e-8b6d-c8960eb95403-kube-api-access-gk99f\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrzp\" (UniqueName: \"kubernetes.io/projected/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-kube-api-access-wwrzp\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.134981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-node-log\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135009 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-cni-netd\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-env-overrides\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/428cf648-de10-4fc7-a55a-ffa7aad31575-agent-certs\") pod \"konnectivity-agent-p5vml\" (UID: \"428cf648-de10-4fc7-a55a-ffa7aad31575\") " pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovnkube-config\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjclj\" (UniqueName: \"kubernetes.io/projected/9423c503-bd8a-4147-9ec9-02137d9fc206-kube-api-access-xjclj\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-systemd-units\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135242 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovnkube-script-lib\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-slash\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-systemd\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-cni-bin\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9423c503-bd8a-4147-9ec9-02137d9fc206-iptables-alerter-script\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135371 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-var-lib-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9423c503-bd8a-4147-9ec9-02137d9fc206-host-slash\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-kubelet\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.135532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135499 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-run-netns\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135525 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-kubelet\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135534 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/428cf648-de10-4fc7-a55a-ffa7aad31575-konnectivity-ca\") pod \"konnectivity-agent-p5vml\" (UID: \"428cf648-de10-4fc7-a55a-ffa7aad31575\") " pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135547 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-ovn\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.134864 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-etc-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135579 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-node-log\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135642 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-cni-netd\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.135650 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-slash\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.135708 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs podName:0d613ce2-7191-434e-8b6d-c8960eb95403 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:17.635689128 +0000 UTC m=+3.074991741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs") pod "network-metrics-daemon-cdtvk" (UID: "0d613ce2-7191-434e-8b6d-c8960eb95403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:17.136242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.135731 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-systemd\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.140697 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-cni-bin\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141226 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-env-overrides\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141244 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovnkube-script-lib\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141341 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141455 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovn-node-metrics-cert\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141578 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-systemd-units\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141668 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-run-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141724 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9423c503-bd8a-4147-9ec9-02137d9fc206-host-slash\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.141776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141764 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9423c503-bd8a-4147-9ec9-02137d9fc206-iptables-alerter-script\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.142267 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.141851 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-var-lib-openvswitch\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.142267 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.140699 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.143061 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.142404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-ovnkube-config\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.146321 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.146294 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/428cf648-de10-4fc7-a55a-ffa7aad31575-agent-certs\") pod \"konnectivity-agent-p5vml\" (UID: \"428cf648-de10-4fc7-a55a-ffa7aad31575\") " pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:17.152223 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.152176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrzp\" (UniqueName: \"kubernetes.io/projected/5157aa65-f232-4a4b-8f18-42cfb4ba47ff-kube-api-access-wwrzp\") pod \"ovnkube-node-jbj8q\" (UID: \"5157aa65-f232-4a4b-8f18-42cfb4ba47ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.170176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.170150 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjclj\" (UniqueName: \"kubernetes.io/projected/9423c503-bd8a-4147-9ec9-02137d9fc206-kube-api-access-xjclj\") pod \"iptables-alerter-xdmmm\" (UID: \"9423c503-bd8a-4147-9ec9-02137d9fc206\") " pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.170678 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.170655 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk99f\" (UniqueName: \"kubernetes.io/projected/0d613ce2-7191-434e-8b6d-c8960eb95403-kube-api-access-gk99f\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:17.214639 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.214603 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4jqnv" Jul 08 19:04:17.221966 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.221941 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" Jul 08 19:04:17.226486 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.226452 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30ca408_d43a_4fc1_bb33_92de7d6fa99d.slice/crio-66bdbe82d7c64d07a2947bc376f2472ae5ebc0dae1462451c1f896244c86ba95 WatchSource:0}: Error finding container 66bdbe82d7c64d07a2947bc376f2472ae5ebc0dae1462451c1f896244c86ba95: Status 404 returned error can't find the container with id 66bdbe82d7c64d07a2947bc376f2472ae5ebc0dae1462451c1f896244c86ba95 Jul 08 19:04:17.230807 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.230784 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df44483_ba6b_41a4_8bfb_e9c7a3f27436.slice/crio-41ae5adca056c18fd599b8a0b21f3df14cca386fc2c73a9b2f3be1dc2531d747 WatchSource:0}: Error finding container 41ae5adca056c18fd599b8a0b21f3df14cca386fc2c73a9b2f3be1dc2531d747: Status 404 returned error can't find the container with id 41ae5adca056c18fd599b8a0b21f3df14cca386fc2c73a9b2f3be1dc2531d747 Jul 08 19:04:17.231544 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.231512 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" Jul 08 19:04:17.236063 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.236043 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rlf8" Jul 08 19:04:17.240183 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.240160 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf794ed16_0d1b_4137_82df_17f6a3830473.slice/crio-5f1dab736f944b203f36ec76c8b834a880a67e83b7f53d288bba6be9c8e0301c WatchSource:0}: Error finding container 5f1dab736f944b203f36ec76c8b834a880a67e83b7f53d288bba6be9c8e0301c: Status 404 returned error can't find the container with id 5f1dab736f944b203f36ec76c8b834a880a67e83b7f53d288bba6be9c8e0301c Jul 08 19:04:17.243963 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.243947 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6jvn" Jul 08 19:04:17.245749 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.245600 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc108ae49_2a6c_44b5_a725_5d3a3b373713.slice/crio-7861d01a148a48f5eb025704b83f87709eba95bdfae23b6c7828112569d43379 WatchSource:0}: Error finding container 7861d01a148a48f5eb025704b83f87709eba95bdfae23b6c7828112569d43379: Status 404 returned error can't find the container with id 7861d01a148a48f5eb025704b83f87709eba95bdfae23b6c7828112569d43379 Jul 08 19:04:17.252651 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.252608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" Jul 08 19:04:17.252965 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.252944 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode154b5fa_dac2_4f7f_bf8b_bdd8b79f8858.slice/crio-89e51a4b7c9bf80f64974518cd50688ffee4759f2f3a0626e24dc046735e1d8e WatchSource:0}: Error finding container 89e51a4b7c9bf80f64974518cd50688ffee4759f2f3a0626e24dc046735e1d8e: Status 404 returned error can't find the container with id 89e51a4b7c9bf80f64974518cd50688ffee4759f2f3a0626e24dc046735e1d8e Jul 08 19:04:17.260152 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.260128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:17.260586 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.260561 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32daff71_c045_4f52_84d2_b4f14425dd63.slice/crio-4cd39cfbcfd19f36a63cfa5b07cdb13789a53b145b2c93a29fb54b8cb4218199 WatchSource:0}: Error finding container 4cd39cfbcfd19f36a63cfa5b07cdb13789a53b145b2c93a29fb54b8cb4218199: Status 404 returned error can't find the container with id 4cd39cfbcfd19f36a63cfa5b07cdb13789a53b145b2c93a29fb54b8cb4218199 Jul 08 19:04:17.267628 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.267424 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdmmm" Jul 08 19:04:17.268390 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.268368 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5157aa65_f232_4a4b_8f18_42cfb4ba47ff.slice/crio-71562385906bdf643a70b21e1c324c032ce7d0b73713272f81bd04069b8b121e WatchSource:0}: Error finding container 71562385906bdf643a70b21e1c324c032ce7d0b73713272f81bd04069b8b121e: Status 404 returned error can't find the container with id 71562385906bdf643a70b21e1c324c032ce7d0b73713272f81bd04069b8b121e Jul 08 19:04:17.273719 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.273695 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:17.276479 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.276458 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9423c503_bd8a_4147_9ec9_02137d9fc206.slice/crio-1e10e20f17cf783614430964be2fd910976675d754ec1898dc1fc81728ef9718 WatchSource:0}: Error finding container 1e10e20f17cf783614430964be2fd910976675d754ec1898dc1fc81728ef9718: Status 404 returned error can't find the container with id 1e10e20f17cf783614430964be2fd910976675d754ec1898dc1fc81728ef9718 Jul 08 19:04:17.286272 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:04:17.286243 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428cf648_de10_4fc7_a55a_ffa7aad31575.slice/crio-c33f85c5f13e45c8d459657f7e384447fcd0008e96b6a5309ca34dc42a29a8c3 WatchSource:0}: Error finding container c33f85c5f13e45c8d459657f7e384447fcd0008e96b6a5309ca34dc42a29a8c3: Status 404 returned error can't find the container with id c33f85c5f13e45c8d459657f7e384447fcd0008e96b6a5309ca34dc42a29a8c3 Jul 08 19:04:17.538956 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.538402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:17.538956 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.538563 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:17.538956 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.538628 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret podName:b1ce6677-e64a-4c21-9faf-a715c7a83d5f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:18.538608532 +0000 UTC m=+3.977911145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret") pod "global-pull-secret-syncer-wc6c2" (UID: "b1ce6677-e64a-4c21-9faf-a715c7a83d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:17.639706 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.639667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:17.639865 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.639736 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:17.639865 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.639844 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:17.639965 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.639901 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs podName:0d613ce2-7191-434e-8b6d-c8960eb95403 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:18.63988374 +0000 UTC m=+4.079186362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs") pod "network-metrics-daemon-cdtvk" (UID: "0d613ce2-7191-434e-8b6d-c8960eb95403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:17.640090 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.639993 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:17.640090 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.640008 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:17.640090 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.640031 2569 projected.go:196] Error preparing data for projected volume kube-api-access-znpkq for pod openshift-network-diagnostics/network-check-target-t9zpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:17.640090 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:17.640068 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq podName:4025f320-f92c-42a0-aa12-69e475af26b9 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:18.640058655 +0000 UTC m=+4.079361278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-znpkq" (UniqueName: "kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq") pod "network-check-target-t9zpq" (UID: "4025f320-f92c-42a0-aa12-69e475af26b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:17.977781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.977730 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:59:17 +0000 UTC" deadline="2028-02-06 02:27:36.206487212 +0000 UTC" Jul 08 19:04:17.978312 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:17.977798 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13855h23m18.228718425s" Jul 08 19:04:18.093218 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.093187 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:18.093390 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.093313 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:18.107981 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.107941 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdmmm" event={"ID":"9423c503-bd8a-4147-9ec9-02137d9fc206","Type":"ContainerStarted","Data":"1e10e20f17cf783614430964be2fd910976675d754ec1898dc1fc81728ef9718"} Jul 08 19:04:18.109725 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.109699 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerStarted","Data":"4cd39cfbcfd19f36a63cfa5b07cdb13789a53b145b2c93a29fb54b8cb4218199"} Jul 08 19:04:18.113535 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.113508 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6jvn" event={"ID":"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858","Type":"ContainerStarted","Data":"89e51a4b7c9bf80f64974518cd50688ffee4759f2f3a0626e24dc046735e1d8e"} Jul 08 19:04:18.119163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.119138 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rlf8" event={"ID":"c108ae49-2a6c-44b5-a725-5d3a3b373713","Type":"ContainerStarted","Data":"7861d01a148a48f5eb025704b83f87709eba95bdfae23b6c7828112569d43379"} Jul 08 19:04:18.121095 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.121073 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" event={"ID":"f794ed16-0d1b-4137-82df-17f6a3830473","Type":"ContainerStarted","Data":"5f1dab736f944b203f36ec76c8b834a880a67e83b7f53d288bba6be9c8e0301c"} Jul 08 19:04:18.126313 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.126272 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" event={"ID":"4df44483-ba6b-41a4-8bfb-e9c7a3f27436","Type":"ContainerStarted","Data":"41ae5adca056c18fd599b8a0b21f3df14cca386fc2c73a9b2f3be1dc2531d747"} Jul 08 19:04:18.130695 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.130669 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4jqnv" event={"ID":"a30ca408-d43a-4fc1-bb33-92de7d6fa99d","Type":"ContainerStarted","Data":"66bdbe82d7c64d07a2947bc376f2472ae5ebc0dae1462451c1f896244c86ba95"} Jul 08 19:04:18.132781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.132706 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p5vml" event={"ID":"428cf648-de10-4fc7-a55a-ffa7aad31575","Type":"ContainerStarted","Data":"c33f85c5f13e45c8d459657f7e384447fcd0008e96b6a5309ca34dc42a29a8c3"} Jul 08 19:04:18.134176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.134152 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"71562385906bdf643a70b21e1c324c032ce7d0b73713272f81bd04069b8b121e"} Jul 08 19:04:18.546451 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.546325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:18.546665 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.546454 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:18.546665 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.546528 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret podName:b1ce6677-e64a-4c21-9faf-a715c7a83d5f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:20.546507878 +0000 UTC m=+5.985810490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret") pod "global-pull-secret-syncer-wc6c2" (UID: "b1ce6677-e64a-4c21-9faf-a715c7a83d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:18.647609 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.647573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:18.647758 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.647634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:18.647809 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.647766 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:18.647858 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.647828 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs podName:0d613ce2-7191-434e-8b6d-c8960eb95403 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:20.647808493 +0000 UTC m=+6.087111107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs") pod "network-metrics-daemon-cdtvk" (UID: "0d613ce2-7191-434e-8b6d-c8960eb95403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:18.648248 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.648227 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:18.648316 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.648253 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:18.648316 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.648265 2569 projected.go:196] Error preparing data for projected volume kube-api-access-znpkq for pod openshift-network-diagnostics/network-check-target-t9zpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:18.648316 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:18.648312 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq podName:4025f320-f92c-42a0-aa12-69e475af26b9 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:20.648297118 +0000 UTC m=+6.087599730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-znpkq" (UniqueName: "kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq") pod "network-check-target-t9zpq" (UID: "4025f320-f92c-42a0-aa12-69e475af26b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:18.978133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.978065 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:59:17 +0000 UTC" deadline="2028-01-20 11:38:31.745748688 +0000 UTC" Jul 08 19:04:18.978133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:18.978131 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13456h34m12.767621446s" Jul 08 19:04:19.094750 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:19.094667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:19.094750 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:19.094709 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:19.094966 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:19.094791 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:19.095210 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:19.095165 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:20.094097 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:20.093862 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:20.094561 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.094178 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:20.155017 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:20.154265 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" event={"ID":"802c451955599a360850d92f832cfcf3","Type":"ContainerStarted","Data":"290be625c335695667067e6abd13da0ec170eb80a9397297d19a9b7c2ed42f38"} Jul 08 19:04:20.172368 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:20.172314 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-33.ec2.internal" podStartSLOduration=4.172293572 podStartE2EDuration="4.172293572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:20.172161746 +0000 UTC m=+5.611464378" watchObservedRunningTime="2026-07-08 19:04:20.172293572 +0000 UTC m=+5.611596206" Jul 08 19:04:20.563874 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:20.563829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:20.564008 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.563982 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:20.564063 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.564031 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret podName:b1ce6677-e64a-4c21-9faf-a715c7a83d5f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:24.564018803 +0000 UTC m=+10.003321425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret") pod "global-pull-secret-syncer-wc6c2" (UID: "b1ce6677-e64a-4c21-9faf-a715c7a83d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:20.664826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:20.664883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.665017 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.665075 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs podName:0d613ce2-7191-434e-8b6d-c8960eb95403 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:24.665057074 +0000 UTC m=+10.104359687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs") pod "network-metrics-daemon-cdtvk" (UID: "0d613ce2-7191-434e-8b6d-c8960eb95403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.665480 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.665496 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.665509 2569 projected.go:196] Error preparing data for projected volume kube-api-access-znpkq for pod openshift-network-diagnostics/network-check-target-t9zpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:20.665607 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:20.665573 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq podName:4025f320-f92c-42a0-aa12-69e475af26b9 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:24.665557886 +0000 UTC m=+10.104860510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-znpkq" (UniqueName: "kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq") pod "network-check-target-t9zpq" (UID: "4025f320-f92c-42a0-aa12-69e475af26b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:21.094270 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:21.093690 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:21.094270 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:21.093722 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:21.094270 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:21.093823 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:21.094270 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:21.094227 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:21.159801 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:21.159734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" event={"ID":"da4443c5c50862dbc167c8fce07acc65","Type":"ContainerStarted","Data":"369f43e7a8d559a8e03e1ff5e984aa9df0d45e6d92a443916bc12eb8a32c1025"} Jul 08 19:04:22.093815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:22.093778 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:22.093984 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:22.093906 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:23.096591 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:23.095568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:23.096591 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:23.095705 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:23.096591 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:23.096128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:23.096591 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:23.096258 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:24.094146 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:24.093700 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:24.094146 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.093829 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:24.598167 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:24.598127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:24.598664 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.598308 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:24.598664 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.598392 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret podName:b1ce6677-e64a-4c21-9faf-a715c7a83d5f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:32.598372523 +0000 UTC m=+18.037675152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret") pod "global-pull-secret-syncer-wc6c2" (UID: "b1ce6677-e64a-4c21-9faf-a715c7a83d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:24.699455 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:24.699421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:24.699641 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:24.699512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:24.699641 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.699608 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:24.699750 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.699656 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:24.699750 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.699677 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:24.699750 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.699684 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs podName:0d613ce2-7191-434e-8b6d-c8960eb95403 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:32.699663597 +0000 UTC m=+18.138966210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs") pod "network-metrics-daemon-cdtvk" (UID: "0d613ce2-7191-434e-8b6d-c8960eb95403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:24.699750 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.699691 2569 projected.go:196] Error preparing data for projected volume kube-api-access-znpkq for pod openshift-network-diagnostics/network-check-target-t9zpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:24.699750 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:24.699739 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq podName:4025f320-f92c-42a0-aa12-69e475af26b9 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:32.699726943 +0000 UTC m=+18.139029564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-znpkq" (UniqueName: "kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq") pod "network-check-target-t9zpq" (UID: "4025f320-f92c-42a0-aa12-69e475af26b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:25.094571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:25.094401 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:25.094571 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:25.094563 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:25.094746 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:25.094603 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:25.094746 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:25.094706 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:25.168649 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:25.168612 2569 generic.go:356] "Generic (PLEG): container finished" podID="da4443c5c50862dbc167c8fce07acc65" containerID="369f43e7a8d559a8e03e1ff5e984aa9df0d45e6d92a443916bc12eb8a32c1025" exitCode=0 Jul 08 19:04:25.168849 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:25.168673 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" event={"ID":"da4443c5c50862dbc167c8fce07acc65","Type":"ContainerDied","Data":"369f43e7a8d559a8e03e1ff5e984aa9df0d45e6d92a443916bc12eb8a32c1025"} Jul 08 19:04:26.093348 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:26.093315 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:26.093776 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:26.093461 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:27.094303 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:27.093675 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:27.094303 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:27.093724 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:27.094303 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:27.093808 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:27.094303 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:27.093912 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:28.094441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:28.093835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:28.094441 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:28.093972 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:28.187460 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:28.187428 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" event={"ID":"f794ed16-0d1b-4137-82df-17f6a3830473","Type":"ContainerStarted","Data":"ba72000069052a1f7871aa645853fec55cc5cf1a358a0faa7ad6b1ac494a8a99"} Jul 08 19:04:28.210159 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:28.208894 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zmxx5" podStartSLOduration=2.448963796 podStartE2EDuration="13.208873634s" totalImagesPullingTime="10.759909838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:28.208457593 +0000 UTC m=+13.647760225" watchObservedRunningTime="2026-07-08 19:04:28.208873634 +0000 UTC m=+13.648176267" Jul 08 19:04:29.093860 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.093668 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:29.094025 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.093692 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:29.094025 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:29.093965 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:29.094148 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:29.094031 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:29.194608 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.194571 2569 generic.go:356] "Generic (PLEG): container finished" podID="32daff71-c045-4f52-84d2-b4f14425dd63" containerID="94fc12b226c4b159b1bc9e463612b8389d15f7f3676156c6ade65d63e1d0735e" exitCode=0 Jul 08 19:04:29.195046 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.194653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerDied","Data":"94fc12b226c4b159b1bc9e463612b8389d15f7f3676156c6ade65d63e1d0735e"} Jul 08 19:04:29.197233 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.197200 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6jvn" event={"ID":"e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858","Type":"ContainerStarted","Data":"3f76a07de8ec9ca4f78f4c9b304cd4eeb907de3e35c7bd97745f46d202a07dca"} Jul 08 19:04:29.199299 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.199271 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rlf8" event={"ID":"c108ae49-2a6c-44b5-a725-5d3a3b373713","Type":"ContainerStarted","Data":"db23e3e24ae8f7c69c6879259c17c5320d297c9878b5b223c9a8d157fe774b7d"} Jul 08 19:04:29.200840 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.200818 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" event={"ID":"4df44483-ba6b-41a4-8bfb-e9c7a3f27436","Type":"ContainerStarted","Data":"b787a7c0aff845c397826bfebeef8dfce2cd6414ba7857f15efa66d40762e7ea"} Jul 08 19:04:29.202795 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.202771 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" event={"ID":"da4443c5c50862dbc167c8fce07acc65","Type":"ContainerStarted","Data":"fc5384afb6cfd1bdb87520dfdefa32196bf58a9cdc7cd2adf72f718198b1a658"} Jul 08 19:04:29.206742 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.206708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p5vml" event={"ID":"428cf648-de10-4fc7-a55a-ffa7aad31575","Type":"ContainerStarted","Data":"1facaca40b85b7378a90d55b85b0fde88e5109d612028edd41b953c4b7a2379f"} Jul 08 19:04:29.253504 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.253457 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l6jvn" podStartSLOduration=3.517203612 podStartE2EDuration="14.25344296s" totalImagesPullingTime="10.736239348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.253309551 +0000 UTC m=+14.692612196" watchObservedRunningTime="2026-07-08 19:04:29.25344296 +0000 UTC m=+14.692745590" Jul 08 19:04:29.272771 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.272639 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p5vml" podStartSLOduration=3.5796013049999997 podStartE2EDuration="14.272618913s" totalImagesPullingTime="10.693017608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.27260913 +0000 UTC m=+14.711911788" watchObservedRunningTime="2026-07-08 19:04:29.272618913 +0000 UTC m=+14.711921538" Jul 08 19:04:29.293114 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.293060 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-33.ec2.internal" podStartSLOduration=13.293042911 podStartE2EDuration="13.293042911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.292626296 +0000 UTC m=+14.731928954" watchObservedRunningTime="2026-07-08 19:04:29.293042911 +0000 UTC m=+14.732345543" Jul 08 19:04:29.311802 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:29.311758 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7rlf8" podStartSLOduration=3.582037212 podStartE2EDuration="14.311745335s" totalImagesPullingTime="10.729708123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:29.310869363 +0000 UTC m=+14.750171996" watchObservedRunningTime="2026-07-08 19:04:29.311745335 +0000 UTC m=+14.751047965" Jul 08 19:04:30.093669 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:30.093633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:30.093871 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:30.093769 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:30.209880 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:30.209840 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdmmm" event={"ID":"9423c503-bd8a-4147-9ec9-02137d9fc206","Type":"ContainerStarted","Data":"609defdd774d4f04caf2c19eca1d17133b72516f1c0336b24c48a6cbcb6f24d1"} Jul 08 19:04:30.227636 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:30.227587 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xdmmm" podStartSLOduration=4.524233645 podStartE2EDuration="15.227571342s" totalImagesPullingTime="10.703337697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:30.227155219 +0000 UTC m=+15.666457870" watchObservedRunningTime="2026-07-08 19:04:30.227571342 +0000 UTC m=+15.666873972" Jul 08 19:04:31.096851 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:31.096357 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:31.096851 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:31.096403 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:31.096851 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:31.096491 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:31.096851 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:31.096625 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:31.459941 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:31.459901 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:31.460508 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:31.460493 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:32.093333 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:32.093301 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:32.093513 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.093419 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:32.214522 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:32.214489 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:32.215125 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:32.215096 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p5vml" Jul 08 19:04:32.324410 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:32.324382 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l6jvn_e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858/dns-node-resolver/0.log" Jul 08 19:04:32.657791 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:32.657763 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:32.658362 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.657909 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:32.658362 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.657983 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret podName:b1ce6677-e64a-4c21-9faf-a715c7a83d5f nodeName:}" failed. No retries permitted until 2026-07-08 19:04:48.657961654 +0000 UTC m=+34.097264288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret") pod "global-pull-secret-syncer-wc6c2" (UID: "b1ce6677-e64a-4c21-9faf-a715c7a83d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:32.758722 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:32.758681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:32.758885 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:32.758747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:32.758885 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.758837 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:32.758885 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.758857 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:32.758885 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.758867 2569 projected.go:196] Error preparing data for projected volume kube-api-access-znpkq for pod openshift-network-diagnostics/network-check-target-t9zpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:32.758885 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.758872 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:32.759071 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.758931 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs podName:0d613ce2-7191-434e-8b6d-c8960eb95403 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:48.758913627 +0000 UTC m=+34.198216256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs") pod "network-metrics-daemon-cdtvk" (UID: "0d613ce2-7191-434e-8b6d-c8960eb95403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:32.759071 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:32.758950 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq podName:4025f320-f92c-42a0-aa12-69e475af26b9 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:48.758939677 +0000 UTC m=+34.198242293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-znpkq" (UniqueName: "kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq") pod "network-check-target-t9zpq" (UID: "4025f320-f92c-42a0-aa12-69e475af26b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:33.093489 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:33.093455 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:33.093671 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:33.093463 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:33.093671 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:33.093603 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:33.093671 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:33.093648 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:33.106634 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:33.106601 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7rlf8_c108ae49-2a6c-44b5-a725-5d3a3b373713/node-ca/0.log" Jul 08 19:04:34.092936 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:34.092898 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:34.093503 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:34.093041 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:35.094226 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:35.094184 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:35.094685 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:35.094297 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:35.094685 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:35.094349 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:35.094685 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:35.094466 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:36.093383 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:36.093342 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:36.093641 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:36.093467 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:37.093963 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:37.093914 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:37.094416 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:37.094073 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:37.094416 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:37.094130 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:37.094416 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:37.094244 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:37.852520 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:37.852346 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 19:04:37.986906 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:37.986811 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T19:04:37.852513468Z","UUID":"54d4e225-5f55-47a7-9353-252c6f4ef641","Handler":null,"Name":"","Endpoint":""} Jul 08 19:04:37.988169 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:37.988151 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 19:04:37.988270 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:37.988189 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 19:04:38.093013 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.092990 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:38.093142 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:38.093116 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:38.229869 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.229633 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" event={"ID":"4df44483-ba6b-41a4-8bfb-e9c7a3f27436","Type":"ContainerStarted","Data":"d88e36e992c5eee855bdc115c15f43c3f266b9abf6a6e71cca0facd3228b7c2d"} Jul 08 19:04:38.231636 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.231605 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4jqnv" event={"ID":"a30ca408-d43a-4fc1-bb33-92de7d6fa99d","Type":"ContainerStarted","Data":"8c4323b6c486a8e3a78ed585f88f600c16a4c9d9da364e5de32d40372c04e44a"} Jul 08 19:04:38.235937 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.235912 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:04:38.236408 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.236380 2569 generic.go:356] "Generic (PLEG): container finished" podID="5157aa65-f232-4a4b-8f18-42cfb4ba47ff" containerID="d16e3fa542d073c85ee5a48037cf1bfa17512b37d8d1357a63b38ec8fe03c5d8" exitCode=1 Jul 08 19:04:38.236497 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.236437 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"0716274776f762e4e29a7f7a2c992608687e302f709ca3b0eb8f0b8c3e5a7d87"} Jul 08 19:04:38.236571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.236500 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"2f051bec93afd6d5d97063cd6b17bd1ab03a433c2d721796b984294edbba67cd"} Jul 08 19:04:38.236626 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.236596 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"7cedac4f37f2b454e89044a34c5c1b788090a4dedc4b5acfcf1fd72a43e34385"} Jul 08 19:04:38.236669 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.236644 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerDied","Data":"d16e3fa542d073c85ee5a48037cf1bfa17512b37d8d1357a63b38ec8fe03c5d8"} Jul 08 19:04:38.236721 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.236688 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"d26ed57cc77c77877920392d268f1d5644457a60c73302e1eb270519c5bba4d2"} Jul 08 19:04:38.251394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:38.251339 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4jqnv" podStartSLOduration=2.81274481 podStartE2EDuration="23.251317479s" totalImagesPullingTime="20.438572669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:38.250418811 +0000 UTC m=+23.689721443" watchObservedRunningTime="2026-07-08 19:04:38.251317479 +0000 UTC m=+23.690620112" Jul 08 19:04:39.093390 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:39.093360 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:39.093651 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:39.093461 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:39.093651 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:39.093489 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:39.093651 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:39.093530 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:39.241415 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:39.241390 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:04:39.241822 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:39.241800 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"d2781f50a2bb7123597e6213c3bde964e261a69ba12390c9f913306571e8c16a"} Jul 08 19:04:40.093196 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:40.093169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:40.093336 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:40.093265 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:40.245765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:40.245714 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" event={"ID":"4df44483-ba6b-41a4-8bfb-e9c7a3f27436","Type":"ContainerStarted","Data":"a72739e686253c8f16bcf3311532c35d9289bc46e944cb74e6a490440ba040d4"} Jul 08 19:04:40.283955 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:40.283907 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nshb" podStartSLOduration=3.68349086 podStartE2EDuration="25.283894114s" totalImagesPullingTime="21.600403254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:40.283617369 +0000 UTC m=+25.722920005" watchObservedRunningTime="2026-07-08 19:04:40.283894114 +0000 UTC m=+25.723196744" Jul 08 19:04:41.093082 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:41.093047 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:41.093440 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:41.093194 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:41.093440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:41.093263 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:41.093440 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:41.093389 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:41.250777 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:41.250751 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:04:41.251185 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:41.251146 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"04bfb7152f26f15ff847c8a1cdf1a14308c50bd54a8cbd6f4d6ac485ae508bf6"} Jul 08 19:04:42.093009 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:42.092971 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:42.093177 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:42.093107 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:43.093484 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:43.093451 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:43.094126 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:43.093498 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:43.094126 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:43.093616 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:43.094126 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:43.093763 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:44.093773 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.093617 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:44.094165 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:44.093849 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:44.261934 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.261909 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:04:44.262269 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.262247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"5d07c0320fc95ba8237ddb356848bdff398ce1ae0d4d4b5e5dc6ad0398701233"} Jul 08 19:04:44.262555 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.262515 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:44.262685 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.262612 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:44.262739 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.262691 2569 scope.go:117] "RemoveContainer" containerID="d16e3fa542d073c85ee5a48037cf1bfa17512b37d8d1357a63b38ec8fe03c5d8" Jul 08 19:04:44.264269 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.264248 2569 generic.go:356] "Generic (PLEG): container finished" podID="32daff71-c045-4f52-84d2-b4f14425dd63" containerID="2634b6c56e27270364622929aefe3bb4d9c36b9afaa5ca5cb65b777b5fefd10c" exitCode=0 Jul 08 19:04:44.264350 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.264292 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerDied","Data":"2634b6c56e27270364622929aefe3bb4d9c36b9afaa5ca5cb65b777b5fefd10c"} Jul 08 19:04:44.280077 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:44.280057 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:45.094508 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.094475 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:45.094971 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:45.094600 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:45.094971 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.094662 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:45.094971 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:45.094785 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:45.268300 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.268267 2569 generic.go:356] "Generic (PLEG): container finished" podID="32daff71-c045-4f52-84d2-b4f14425dd63" containerID="46a6b5c549f498565dc80f0dd3ec90c6db23315163d8b32335ffdc1b8a982c2d" exitCode=0 Jul 08 19:04:45.268411 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.268356 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerDied","Data":"46a6b5c549f498565dc80f0dd3ec90c6db23315163d8b32335ffdc1b8a982c2d"} Jul 08 19:04:45.272137 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.272121 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:04:45.272450 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.272425 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" event={"ID":"5157aa65-f232-4a4b-8f18-42cfb4ba47ff","Type":"ContainerStarted","Data":"09388905ccc0edf36070b9aaef34e9be83536217adfd6813380c925c0fd1517a"} Jul 08 19:04:45.272714 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.272699 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:45.288351 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.288329 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:04:45.341350 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:45.341311 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" podStartSLOduration=9.894864958 podStartE2EDuration="30.34129846s" totalImagesPullingTime="20.446433502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:45.341191318 +0000 UTC m=+30.780493949" watchObservedRunningTime="2026-07-08 19:04:45.34129846 +0000 UTC m=+30.780601091" Jul 08 19:04:46.093858 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:46.093827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:46.093977 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:46.093925 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:46.277708 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:46.277625 2569 generic.go:356] "Generic (PLEG): container finished" podID="32daff71-c045-4f52-84d2-b4f14425dd63" containerID="a0604f6c63bde0fa93502c0c7713e13e8d1bcec81373cf0c5e32bff876b8b863" exitCode=0 Jul 08 19:04:46.277708 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:46.277697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerDied","Data":"a0604f6c63bde0fa93502c0c7713e13e8d1bcec81373cf0c5e32bff876b8b863"} Jul 08 19:04:47.093185 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:47.093137 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:47.093354 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:47.093243 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:47.093354 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:47.093267 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:47.093354 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:47.093342 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:48.093765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:48.093728 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:48.094147 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.093852 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:48.670210 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:48.670174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:48.670470 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.670339 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:48.670470 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.670416 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret podName:b1ce6677-e64a-4c21-9faf-a715c7a83d5f nodeName:}" failed. No retries permitted until 2026-07-08 19:05:20.670395693 +0000 UTC m=+66.109698317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret") pod "global-pull-secret-syncer-wc6c2" (UID: "b1ce6677-e64a-4c21-9faf-a715c7a83d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:04:48.771406 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:48.771373 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:48.771406 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:48.771412 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:48.771681 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.771512 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:48.771681 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.771564 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:04:48.771681 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.771587 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:04:48.771681 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.771591 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs podName:0d613ce2-7191-434e-8b6d-c8960eb95403 nodeName:}" failed. No retries permitted until 2026-07-08 19:05:20.771574341 +0000 UTC m=+66.210876970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs") pod "network-metrics-daemon-cdtvk" (UID: "0d613ce2-7191-434e-8b6d-c8960eb95403") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:04:48.771681 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.771599 2569 projected.go:196] Error preparing data for projected volume kube-api-access-znpkq for pod openshift-network-diagnostics/network-check-target-t9zpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:48.771681 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:48.771651 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq podName:4025f320-f92c-42a0-aa12-69e475af26b9 nodeName:}" failed. No retries permitted until 2026-07-08 19:05:20.771635043 +0000 UTC m=+66.210937661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-znpkq" (UniqueName: "kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq") pod "network-check-target-t9zpq" (UID: "4025f320-f92c-42a0-aa12-69e475af26b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:04:49.093707 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:49.093617 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:49.093707 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:49.093644 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:49.094205 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:49.093719 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:49.094205 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:49.093848 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:50.093043 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:50.093011 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:50.093208 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:50.093139 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:51.093912 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:51.093874 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:51.093912 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:51.093895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:51.094426 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:51.094014 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:51.094426 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:51.094160 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:52.092947 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:52.092865 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:52.093088 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:52.092964 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:52.292417 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:52.292388 2569 generic.go:356] "Generic (PLEG): container finished" podID="32daff71-c045-4f52-84d2-b4f14425dd63" containerID="64f9bc32fcf1f70db1d4df3e324c7e0ba29626f6e234aeffb2db09793db31d38" exitCode=0 Jul 08 19:04:52.292770 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:52.292445 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerDied","Data":"64f9bc32fcf1f70db1d4df3e324c7e0ba29626f6e234aeffb2db09793db31d38"} Jul 08 19:04:53.093845 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:53.093815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:53.094022 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:53.093815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:53.094022 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:53.093918 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:53.094022 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:53.093999 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:53.297591 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:53.297558 2569 generic.go:356] "Generic (PLEG): container finished" podID="32daff71-c045-4f52-84d2-b4f14425dd63" containerID="060db191e586166132d94921a95aabc7cc909c41d3acf75863f54d8e206c9dd8" exitCode=0 Jul 08 19:04:53.297591 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:53.297570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerDied","Data":"060db191e586166132d94921a95aabc7cc909c41d3acf75863f54d8e206c9dd8"} Jul 08 19:04:54.093297 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:54.093264 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:54.093473 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:54.093377 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:54.303085 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:54.303043 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" event={"ID":"32daff71-c045-4f52-84d2-b4f14425dd63","Type":"ContainerStarted","Data":"506bde311361781809686d8ad8a8dc9653a7a614b072eb6f626cffa951f8742d"} Jul 08 19:04:54.330169 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:54.330117 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vqsjw" podStartSLOduration=6.960625279 podStartE2EDuration="39.330100732s" totalImagesPullingTime="32.369475453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:54.329570009 +0000 UTC m=+39.768872659" watchObservedRunningTime="2026-07-08 19:04:54.330100732 +0000 UTC m=+39.769403363" Jul 08 19:04:55.098133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.098088 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:55.098331 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.098134 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:55.098331 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:55.098192 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:55.098331 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:55.098271 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:55.832362 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.832193 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t9zpq"] Jul 08 19:04:55.832980 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.832437 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:55.832980 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:55.832567 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:55.834494 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.834467 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cdtvk"] Jul 08 19:04:55.834628 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.834607 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:55.834743 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:55.834721 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:55.835138 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.835092 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wc6c2"] Jul 08 19:04:55.835248 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:55.835216 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:55.835329 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:55.835309 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:57.093128 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:57.093102 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:57.093566 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:57.093214 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:04:58.093502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:58.093466 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:04:58.093894 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:58.093469 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:04:58.093894 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:58.093591 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:04:58.093894 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:58.093663 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:04:59.093183 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:04:59.093156 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:04:59.093345 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:04:59.093271 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cdtvk" podUID="0d613ce2-7191-434e-8b6d-c8960eb95403" Jul 08 19:05:00.093581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.093559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:05:00.093834 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.093565 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:05:00.093834 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:00.093661 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t9zpq" podUID="4025f320-f92c-42a0-aa12-69e475af26b9" Jul 08 19:05:00.093834 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:00.093730 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc6c2" podUID="b1ce6677-e64a-4c21-9faf-a715c7a83d5f" Jul 08 19:05:00.297879 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.297845 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-33.ec2.internal" event="NodeReady" Jul 08 19:05:00.298057 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.297963 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 19:05:00.346152 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.346067 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w8k79"] Jul 08 19:05:00.351001 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.350980 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vj2f5"] Jul 08 19:05:00.351140 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.351117 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.353705 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.353685 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 19:05:00.353786 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.353733 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 19:05:00.353823 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.353807 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xm8m6\"" Jul 08 19:05:00.357507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.357233 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.359757 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.359737 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 19:05:00.360063 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.360041 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5l8cc\"" Jul 08 19:05:00.360170 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.360083 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 19:05:00.360407 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.360389 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8k79"] Jul 08 19:05:00.360594 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.360574 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 19:05:00.363532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.363514 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vj2f5"] Jul 08 19:05:00.466579 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.466534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dvz\" (UniqueName: \"kubernetes.io/projected/684f9a0f-00ac-47cf-884c-84d79e10e510-kube-api-access-t6dvz\") pod \"ingress-canary-vj2f5\" (UID: \"684f9a0f-00ac-47cf-884c-84d79e10e510\") " pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.466755 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.466605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f81f8d7-4135-4fc0-ab60-95701bca82d4-tmp-dir\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.466755 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.466633 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f81f8d7-4135-4fc0-ab60-95701bca82d4-metrics-tls\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.466755 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.466654 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f81f8d7-4135-4fc0-ab60-95701bca82d4-config-volume\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.466755 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.466674 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/684f9a0f-00ac-47cf-884c-84d79e10e510-cert\") pod \"ingress-canary-vj2f5\" (UID: \"684f9a0f-00ac-47cf-884c-84d79e10e510\") " pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.466935 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.466810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7zr\" (UniqueName: \"kubernetes.io/projected/4f81f8d7-4135-4fc0-ab60-95701bca82d4-kube-api-access-qb7zr\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.470842 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.470816 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4bg6"] Jul 08 19:05:00.487999 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.487974 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4bg6"] Jul 08 19:05:00.488127 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.488096 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.490757 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.490731 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 19:05:00.490757 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.490731 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 19:05:00.490982 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.490787 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6ts4z\"" Jul 08 19:05:00.490982 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.490737 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 19:05:00.491135 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.491053 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 19:05:00.567533 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.567500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dvz\" (UniqueName: \"kubernetes.io/projected/684f9a0f-00ac-47cf-884c-84d79e10e510-kube-api-access-t6dvz\") pod \"ingress-canary-vj2f5\" (UID: \"684f9a0f-00ac-47cf-884c-84d79e10e510\") " pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.567695 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.567576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f81f8d7-4135-4fc0-ab60-95701bca82d4-tmp-dir\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.567695 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.567597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f81f8d7-4135-4fc0-ab60-95701bca82d4-metrics-tls\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.567695 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.567617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f81f8d7-4135-4fc0-ab60-95701bca82d4-config-volume\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.567695 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.567639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/684f9a0f-00ac-47cf-884c-84d79e10e510-cert\") pod \"ingress-canary-vj2f5\" (UID: \"684f9a0f-00ac-47cf-884c-84d79e10e510\") " pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.567912 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.567755 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7zr\" (UniqueName: \"kubernetes.io/projected/4f81f8d7-4135-4fc0-ab60-95701bca82d4-kube-api-access-qb7zr\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.568040 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.568019 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f81f8d7-4135-4fc0-ab60-95701bca82d4-tmp-dir\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.568340 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.568317 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f81f8d7-4135-4fc0-ab60-95701bca82d4-config-volume\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.571675 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.571657 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f81f8d7-4135-4fc0-ab60-95701bca82d4-metrics-tls\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.571774 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.571757 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/684f9a0f-00ac-47cf-884c-84d79e10e510-cert\") pod \"ingress-canary-vj2f5\" (UID: \"684f9a0f-00ac-47cf-884c-84d79e10e510\") " pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.576119 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.576096 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7zr\" (UniqueName: \"kubernetes.io/projected/4f81f8d7-4135-4fc0-ab60-95701bca82d4-kube-api-access-qb7zr\") pod \"dns-default-w8k79\" (UID: \"4f81f8d7-4135-4fc0-ab60-95701bca82d4\") " pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.576642 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.576621 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dvz\" (UniqueName: \"kubernetes.io/projected/684f9a0f-00ac-47cf-884c-84d79e10e510-kube-api-access-t6dvz\") pod \"ingress-canary-vj2f5\" (UID: \"684f9a0f-00ac-47cf-884c-84d79e10e510\") " pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.659946 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.659868 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:00.668007 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.667979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vj2f5" Jul 08 19:05:00.668149 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.668128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn976\" (UniqueName: \"kubernetes.io/projected/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-kube-api-access-wn976\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.668189 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.668160 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-crio-socket\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.668189 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.668177 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.668259 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.668218 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-data-volume\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.668304 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.668260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.769323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.769640 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn976\" (UniqueName: \"kubernetes.io/projected/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-kube-api-access-wn976\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.769681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-crio-socket\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.769707 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.769739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-data-volume\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.770070 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-data-volume\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.770219 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-crio-socket\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.770524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.770070 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.773706 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.773680 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.778168 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.778147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn976\" (UniqueName: \"kubernetes.io/projected/c89e967c-2e64-43c4-ba0e-ac8c221bb8f7-kube-api-access-wn976\") pod \"insights-runtime-extractor-s4bg6\" (UID: \"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7\") " pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.796624 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.796595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4bg6" Jul 08 19:05:00.842489 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.842459 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vj2f5"] Jul 08 19:05:00.845854 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.845564 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8k79"] Jul 08 19:05:00.848088 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:00.848060 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684f9a0f_00ac_47cf_884c_84d79e10e510.slice/crio-2a493f3bbb4b08ba7a29f62a25a7ec9bcd5bbb041336fdd8c5d688f8082977d5 WatchSource:0}: Error finding container 2a493f3bbb4b08ba7a29f62a25a7ec9bcd5bbb041336fdd8c5d688f8082977d5: Status 404 returned error can't find the container with id 2a493f3bbb4b08ba7a29f62a25a7ec9bcd5bbb041336fdd8c5d688f8082977d5 Jul 08 19:05:00.849803 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:00.849777 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f81f8d7_4135_4fc0_ab60_95701bca82d4.slice/crio-505f8645012a29e1534e7c60a7b7d235f011175a96282b140cfa3966f165c2d2 WatchSource:0}: Error finding container 505f8645012a29e1534e7c60a7b7d235f011175a96282b140cfa3966f165c2d2: Status 404 returned error can't find the container with id 505f8645012a29e1534e7c60a7b7d235f011175a96282b140cfa3966f165c2d2 Jul 08 19:05:00.946443 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:00.946416 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4bg6"] Jul 08 19:05:00.949611 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:00.949585 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89e967c_2e64_43c4_ba0e_ac8c221bb8f7.slice/crio-a5e546b422679c8d4be765bf6d347c32714b9dde5e33f281dfa7fde84b75c321 WatchSource:0}: Error finding container a5e546b422679c8d4be765bf6d347c32714b9dde5e33f281dfa7fde84b75c321: Status 404 returned error can't find the container with id a5e546b422679c8d4be765bf6d347c32714b9dde5e33f281dfa7fde84b75c321 Jul 08 19:05:01.093599 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.093566 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:05:01.096571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.096532 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kpjrj\"" Jul 08 19:05:01.096714 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.096554 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:05:01.320724 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.320638 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4bg6" event={"ID":"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7","Type":"ContainerStarted","Data":"a5e546b422679c8d4be765bf6d347c32714b9dde5e33f281dfa7fde84b75c321"} Jul 08 19:05:01.321740 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.321714 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vj2f5" event={"ID":"684f9a0f-00ac-47cf-884c-84d79e10e510","Type":"ContainerStarted","Data":"2a493f3bbb4b08ba7a29f62a25a7ec9bcd5bbb041336fdd8c5d688f8082977d5"} Jul 08 19:05:01.322751 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.322723 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8k79" event={"ID":"4f81f8d7-4135-4fc0-ab60-95701bca82d4","Type":"ContainerStarted","Data":"505f8645012a29e1534e7c60a7b7d235f011175a96282b140cfa3966f165c2d2"} Jul 08 19:05:01.399482 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.399454 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9"] Jul 08 19:05:01.415769 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.415748 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9"] Jul 08 19:05:01.415877 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.415844 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.418560 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.418518 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 08 19:05:01.418666 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.418572 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 19:05:01.419020 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.419001 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4kp2n\"" Jul 08 19:05:01.419748 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.419716 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 19:05:01.419748 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.419736 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 08 19:05:01.419892 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.419738 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 19:05:01.575620 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.575529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.575777 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.575718 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4f38d4c-09e7-427f-8f02-acf137525200-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.575777 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.575757 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgfl\" (UniqueName: \"kubernetes.io/projected/a4f38d4c-09e7-427f-8f02-acf137525200-kube-api-access-fwgfl\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.575885 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.575789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.676647 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.676609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.676837 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.676737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4f38d4c-09e7-427f-8f02-acf137525200-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.676837 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.676757 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgfl\" (UniqueName: \"kubernetes.io/projected/a4f38d4c-09e7-427f-8f02-acf137525200-kube-api-access-fwgfl\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.676837 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.676781 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.677585 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:01.677138 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 08 19:05:01.677585 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:01.677219 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls podName:a4f38d4c-09e7-427f-8f02-acf137525200 nodeName:}" failed. No retries permitted until 2026-07-08 19:05:02.177197552 +0000 UTC m=+47.616500177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-9ccf9" (UID: "a4f38d4c-09e7-427f-8f02-acf137525200") : secret "prometheus-operator-tls" not found Jul 08 19:05:01.678031 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.677978 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4f38d4c-09e7-427f-8f02-acf137525200-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.680651 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.680626 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:01.695063 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:01.694988 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgfl\" (UniqueName: \"kubernetes.io/projected/a4f38d4c-09e7-427f-8f02-acf137525200-kube-api-access-fwgfl\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:02.092919 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:02.092884 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:05:02.093400 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:02.093380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:05:02.096134 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:02.096099 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:05:02.096915 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:02.096894 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:05:02.097015 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:02.096903 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gzqsf\"" Jul 08 19:05:02.097015 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:02.096943 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:05:02.181870 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:02.181767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:02.182045 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:02.181949 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 08 19:05:02.182121 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:02.182107 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls podName:a4f38d4c-09e7-427f-8f02-acf137525200 nodeName:}" failed. No retries permitted until 2026-07-08 19:05:03.182086583 +0000 UTC m=+48.621389207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-9ccf9" (UID: "a4f38d4c-09e7-427f-8f02-acf137525200") : secret "prometheus-operator-tls" not found Jul 08 19:05:03.191825 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:03.191785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:03.194927 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:03.194899 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4f38d4c-09e7-427f-8f02-acf137525200-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-9ccf9\" (UID: \"a4f38d4c-09e7-427f-8f02-acf137525200\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:03.224798 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:03.224762 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" Jul 08 19:05:07.334516 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.334491 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9"] Jul 08 19:05:07.336979 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:07.336954 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f38d4c_09e7_427f_8f02_acf137525200.slice/crio-39810749a8002ea19ecdc08c6e38e9e32eb6e549c9acbf47546172f065507aff WatchSource:0}: Error finding container 39810749a8002ea19ecdc08c6e38e9e32eb6e549c9acbf47546172f065507aff: Status 404 returned error can't find the container with id 39810749a8002ea19ecdc08c6e38e9e32eb6e549c9acbf47546172f065507aff Jul 08 19:05:07.340237 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.340216 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8k79" event={"ID":"4f81f8d7-4135-4fc0-ab60-95701bca82d4","Type":"ContainerStarted","Data":"7e19511c1fd5dd4635bd758837cbba7dcf726defb0e2a561447b2c028239b86c"} Jul 08 19:05:07.933435 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.933402 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-vw4jc"] Jul 08 19:05:07.956479 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.956455 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-vw4jc" Jul 08 19:05:07.965070 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.965052 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cj9jq\"" Jul 08 19:05:07.966999 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.966986 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 19:05:07.967206 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.967195 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 19:05:07.982581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:07.982552 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-vw4jc"] Jul 08 19:05:08.031512 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.031479 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsnm2\" (UniqueName: \"kubernetes.io/projected/164eba15-890e-4e1b-a0e6-e7750d37db3d-kube-api-access-wsnm2\") pod \"downloads-685bfb5f64-vw4jc\" (UID: \"164eba15-890e-4e1b-a0e6-e7750d37db3d\") " pod="openshift-console/downloads-685bfb5f64-vw4jc" Jul 08 19:05:08.132144 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.132113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsnm2\" (UniqueName: \"kubernetes.io/projected/164eba15-890e-4e1b-a0e6-e7750d37db3d-kube-api-access-wsnm2\") pod \"downloads-685bfb5f64-vw4jc\" (UID: \"164eba15-890e-4e1b-a0e6-e7750d37db3d\") " pod="openshift-console/downloads-685bfb5f64-vw4jc" Jul 08 19:05:08.146149 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.146122 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsnm2\" (UniqueName: \"kubernetes.io/projected/164eba15-890e-4e1b-a0e6-e7750d37db3d-kube-api-access-wsnm2\") pod \"downloads-685bfb5f64-vw4jc\" (UID: \"164eba15-890e-4e1b-a0e6-e7750d37db3d\") " pod="openshift-console/downloads-685bfb5f64-vw4jc" Jul 08 19:05:08.264338 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.264261 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-vw4jc" Jul 08 19:05:08.344944 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.344902 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4bg6" event={"ID":"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7","Type":"ContainerStarted","Data":"f67cf74165c4f3624790d26639a6e347294c14507420c8c91304ffcc764024f4"} Jul 08 19:05:08.346937 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.346898 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8k79" event={"ID":"4f81f8d7-4135-4fc0-ab60-95701bca82d4","Type":"ContainerStarted","Data":"35954adac6665e757e75b419c77d1d34b4f505f6926836b3d6a1aa21cad7467d"} Jul 08 19:05:08.347072 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.347041 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:08.348519 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.348496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vj2f5" event={"ID":"684f9a0f-00ac-47cf-884c-84d79e10e510","Type":"ContainerStarted","Data":"5990f6b6b0728c161ee4899eb08556de1da256978d3a520adae19a3671082643"} Jul 08 19:05:08.349656 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.349628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" event={"ID":"a4f38d4c-09e7-427f-8f02-acf137525200","Type":"ContainerStarted","Data":"39810749a8002ea19ecdc08c6e38e9e32eb6e549c9acbf47546172f065507aff"} Jul 08 19:05:08.370392 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.370347 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w8k79" podStartSLOduration=2.040446355 podStartE2EDuration="8.370331616s" totalImagesPullingTime="6.329885261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:08.369951271 +0000 UTC m=+53.809253903" watchObservedRunningTime="2026-07-08 19:05:08.370331616 +0000 UTC m=+53.809634247" Jul 08 19:05:08.391431 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.391388 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vj2f5" podStartSLOduration=2.051919134 podStartE2EDuration="8.391372076s" totalImagesPullingTime="6.339452942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:08.390787996 +0000 UTC m=+53.830090630" watchObservedRunningTime="2026-07-08 19:05:08.391372076 +0000 UTC m=+53.830674706" Jul 08 19:05:08.544868 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:08.544795 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-vw4jc"] Jul 08 19:05:08.547294 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:08.547268 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164eba15_890e_4e1b_a0e6_e7750d37db3d.slice/crio-d48eaa9f8ac2fd1334ee6ae047bacd0cd4f05b55f4bee07f5e7b55e41fdb99b5 WatchSource:0}: Error finding container d48eaa9f8ac2fd1334ee6ae047bacd0cd4f05b55f4bee07f5e7b55e41fdb99b5: Status 404 returned error can't find the container with id d48eaa9f8ac2fd1334ee6ae047bacd0cd4f05b55f4bee07f5e7b55e41fdb99b5 Jul 08 19:05:09.356356 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:09.356315 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4bg6" event={"ID":"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7","Type":"ContainerStarted","Data":"01f5e6341c023e3a8a7c1c650880f708146c5cb23dbb518f9e81d3fc119649ee"} Jul 08 19:05:09.358030 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:09.358000 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-vw4jc" event={"ID":"164eba15-890e-4e1b-a0e6-e7750d37db3d","Type":"ContainerStarted","Data":"d48eaa9f8ac2fd1334ee6ae047bacd0cd4f05b55f4bee07f5e7b55e41fdb99b5"} Jul 08 19:05:10.365101 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:10.365021 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" event={"ID":"a4f38d4c-09e7-427f-8f02-acf137525200","Type":"ContainerStarted","Data":"b1cf242f9a9ba5637cbcc71601a5b0485aaa30d4314751534d219617f1eb0c21"} Jul 08 19:05:10.365101 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:10.365082 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" event={"ID":"a4f38d4c-09e7-427f-8f02-acf137525200","Type":"ContainerStarted","Data":"f1ebb85a8fa001e33f44fbb875a71ec231efc92f7347b5aacb3024ebabe82557"} Jul 08 19:05:10.384574 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:10.384505 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-9ccf9" podStartSLOduration=6.7472709 podStartE2EDuration="9.384485445s" totalImagesPullingTime="2.637214545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:10.384098962 +0000 UTC m=+55.823401620" watchObservedRunningTime="2026-07-08 19:05:10.384485445 +0000 UTC m=+55.823788079" Jul 08 19:05:12.375073 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.375025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4bg6" event={"ID":"c89e967c-2e64-43c4-ba0e-ac8c221bb8f7","Type":"ContainerStarted","Data":"581d206471548ae56db6ee3f33ee956b5c0e18b4b28c57c664fd24bd6c3b3f50"} Jul 08 19:05:12.396620 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.396574 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s4bg6" podStartSLOduration=1.746637066 podStartE2EDuration="12.396558934s" totalImagesPullingTime="10.649921868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:05:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:12.395849437 +0000 UTC m=+57.835152069" watchObservedRunningTime="2026-07-08 19:05:12.396558934 +0000 UTC m=+57.835861555" Jul 08 19:05:12.746791 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.746759 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz"] Jul 08 19:05:12.748982 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.748956 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.749719 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.749692 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r"] Jul 08 19:05:12.751519 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.751489 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 08 19:05:12.751636 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.751518 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 08 19:05:12.751636 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.751562 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-794vq\"" Jul 08 19:05:12.751915 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.751899 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.754392 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.754376 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 08 19:05:12.754520 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.754504 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bjks2\"" Jul 08 19:05:12.754731 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.754700 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 08 19:05:12.754837 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.754782 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 08 19:05:12.762573 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.762528 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz"] Jul 08 19:05:12.765103 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.764939 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.765103 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.764975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.765103 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.765103 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e3d39085-2b8b-4ca0-a91c-93912b9a847a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.765103 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.765394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.765394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765148 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xr6l\" (UniqueName: \"kubernetes.io/projected/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-api-access-9xr6l\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.765394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67xb\" (UniqueName: \"kubernetes.io/projected/9989c28e-0ee9-4684-a305-152f684d7a5d-kube-api-access-v67xb\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.765394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765204 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d39085-2b8b-4ca0-a91c-93912b9a847a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.765394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.765226 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9989c28e-0ee9-4684-a305-152f684d7a5d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.769079 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.769056 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r"] Jul 08 19:05:12.777099 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.777081 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9tr92"] Jul 08 19:05:12.779245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.779227 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.781810 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.781780 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h87lk\"" Jul 08 19:05:12.781923 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.781907 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 19:05:12.782020 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.782007 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 19:05:12.782116 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.782101 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 19:05:12.865583 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.865583 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.865813 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e3d39085-2b8b-4ca0-a91c-93912b9a847a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.865813 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865641 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-root\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.865813 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-textfile\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.865813 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xr6l\" (UniqueName: \"kubernetes.io/projected/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-api-access-9xr6l\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.865955 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67xb\" (UniqueName: \"kubernetes.io/projected/9989c28e-0ee9-4684-a305-152f684d7a5d-kube-api-access-v67xb\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.865955 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4d7\" (UniqueName: \"kubernetes.io/projected/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-kube-api-access-wb4d7\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.866031 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d39085-2b8b-4ca0-a91c-93912b9a847a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.866031 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.865992 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e3d39085-2b8b-4ca0-a91c-93912b9a847a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.866031 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-sys\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.866136 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.866136 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866092 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.866212 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-tls\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.866212 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.866212 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-metrics-client-ca\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.866360 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866275 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.866360 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9989c28e-0ee9-4684-a305-152f684d7a5d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.866360 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866323 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.866360 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:12.866344 2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 08 19:05:12.866360 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-wtmp\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.866643 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:12.866415 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 08 19:05:12.866643 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:12.866433 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-tls podName:9989c28e-0ee9-4684-a305-152f684d7a5d nodeName:}" failed. No retries permitted until 2026-07-08 19:05:13.36641393 +0000 UTC m=+58.805716558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-24sdz" (UID: "9989c28e-0ee9-4684-a305-152f684d7a5d") : secret "openshift-state-metrics-tls" not found Jul 08 19:05:12.866643 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.866643 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:12.866473 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-tls podName:e3d39085-2b8b-4ca0-a91c-93912b9a847a nodeName:}" failed. No retries permitted until 2026-07-08 19:05:13.366456668 +0000 UTC m=+58.805759294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-sv47r" (UID: "e3d39085-2b8b-4ca0-a91c-93912b9a847a") : secret "kube-state-metrics-tls" not found Jul 08 19:05:12.866643 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866572 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d39085-2b8b-4ca0-a91c-93912b9a847a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.867071 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.866997 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9989c28e-0ee9-4684-a305-152f684d7a5d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.868525 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.868504 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.868525 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.868523 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.875580 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.875532 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67xb\" (UniqueName: \"kubernetes.io/projected/9989c28e-0ee9-4684-a305-152f684d7a5d-kube-api-access-v67xb\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:12.875580 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.875573 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xr6l\" (UniqueName: \"kubernetes.io/projected/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-api-access-9xr6l\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:12.967214 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4d7\" (UniqueName: \"kubernetes.io/projected/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-kube-api-access-wb4d7\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967214 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-sys\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967415 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967283 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-sys\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967415 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967356 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-tls\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967415 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-metrics-client-ca\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967519 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-wtmp\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967519 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:12.967449 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 19:05:12.967519 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967456 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967653 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967653 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:12.967590 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-tls podName:0e30e11f-9fec-4d54-8f79-01da25f8b3ae nodeName:}" failed. No retries permitted until 2026-07-08 19:05:13.46756955 +0000 UTC m=+58.906872163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-tls") pod "node-exporter-9tr92" (UID: "0e30e11f-9fec-4d54-8f79-01da25f8b3ae") : secret "node-exporter-tls" not found Jul 08 19:05:12.967653 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967619 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-wtmp\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-root\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967687 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-textfile\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.967765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.967704 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-root\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.968020 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.968005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-textfile\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.968073 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.968044 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.968138 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.968120 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-metrics-client-ca\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.970461 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.970436 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:12.978902 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:12.978874 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4d7\" (UniqueName: \"kubernetes.io/projected/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-kube-api-access-wb4d7\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:13.372701 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.372665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:13.372866 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.372718 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:13.375229 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.375202 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3d39085-2b8b-4ca0-a91c-93912b9a847a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-sv47r\" (UID: \"e3d39085-2b8b-4ca0-a91c-93912b9a847a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:13.375571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.375235 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9989c28e-0ee9-4684-a305-152f684d7a5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-24sdz\" (UID: \"9989c28e-0ee9-4684-a305-152f684d7a5d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:13.473756 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.473719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-tls\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:13.476032 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.476015 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e30e11f-9fec-4d54-8f79-01da25f8b3ae-node-exporter-tls\") pod \"node-exporter-9tr92\" (UID: \"0e30e11f-9fec-4d54-8f79-01da25f8b3ae\") " pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:13.661310 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.661281 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" Jul 08 19:05:13.668965 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.668944 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" Jul 08 19:05:13.690888 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.690860 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tr92" Jul 08 19:05:13.702250 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:13.702097 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e30e11f_9fec_4d54_8f79_01da25f8b3ae.slice/crio-8668639e2997e4012b1f8c78c0e34ae03c61d10facdf6020f5f3ad11a859874a WatchSource:0}: Error finding container 8668639e2997e4012b1f8c78c0e34ae03c61d10facdf6020f5f3ad11a859874a: Status 404 returned error can't find the container with id 8668639e2997e4012b1f8c78c0e34ae03c61d10facdf6020f5f3ad11a859874a Jul 08 19:05:13.813780 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.813749 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz"] Jul 08 19:05:13.817666 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:13.817635 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9989c28e_0ee9_4684_a305_152f684d7a5d.slice/crio-84b7b83dc8277157cb4408657fdc233e205b05422013f48b9f8cf8dd722c6a9b WatchSource:0}: Error finding container 84b7b83dc8277157cb4408657fdc233e205b05422013f48b9f8cf8dd722c6a9b: Status 404 returned error can't find the container with id 84b7b83dc8277157cb4408657fdc233e205b05422013f48b9f8cf8dd722c6a9b Jul 08 19:05:13.843790 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.843764 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r"] Jul 08 19:05:13.850697 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:13.850671 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d39085_2b8b_4ca0_a91c_93912b9a847a.slice/crio-77a67b48fca513bcd38f661fd3ad985b284a405d015feb445741e74f981ba991 WatchSource:0}: Error finding container 77a67b48fca513bcd38f661fd3ad985b284a405d015feb445741e74f981ba991: Status 404 returned error can't find the container with id 77a67b48fca513bcd38f661fd3ad985b284a405d015feb445741e74f981ba991 Jul 08 19:05:13.885361 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.884679 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:05:13.887715 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.887694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.890524 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.890856 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.890982 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.891152 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.891458 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.891589 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tc6fb\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.891774 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.893231 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 19:05:13.893690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.893572 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 19:05:13.894185 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.893961 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 19:05:13.913211 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.913154 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978530 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-config-volume\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn67\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-kube-api-access-9jn67\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978787 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-config-out\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978843 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-web-config\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978886 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978939 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:13.979124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:13.978971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080337 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.079835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-config-volume\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080521 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080521 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn67\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-kube-api-access-9jn67\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080666 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080604 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080666 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080775 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-config-out\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080878 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080859 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080935 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080889 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-web-config\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.080935 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.080913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.081279 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.081246 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.081394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.081301 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.081457 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.081426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.081509 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.081461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.081588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.081520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.081737 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.081719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.081858 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:14.081841 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 08 19:05:14.082074 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:14.081908 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls podName:84f99285-4558-48e6-b72d-a237e32d368e nodeName:}" failed. No retries permitted until 2026-07-08 19:05:14.581890213 +0000 UTC m=+60.021192823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "84f99285-4558-48e6-b72d-a237e32d368e") : secret "alertmanager-main-tls" not found Jul 08 19:05:14.083145 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.083125 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-config-volume\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.083354 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.083331 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.084444 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.084398 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-config-out\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.084955 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.084934 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.085049 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.085027 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.085457 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.085438 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.085948 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.085926 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-web-config\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.086248 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.086226 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.086924 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.086905 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.090111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.090086 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn67\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-kube-api-access-9jn67\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.384353 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.384272 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tr92" event={"ID":"0e30e11f-9fec-4d54-8f79-01da25f8b3ae","Type":"ContainerStarted","Data":"8668639e2997e4012b1f8c78c0e34ae03c61d10facdf6020f5f3ad11a859874a"} Jul 08 19:05:14.385752 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.385720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" event={"ID":"e3d39085-2b8b-4ca0-a91c-93912b9a847a","Type":"ContainerStarted","Data":"77a67b48fca513bcd38f661fd3ad985b284a405d015feb445741e74f981ba991"} Jul 08 19:05:14.387803 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.387776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" event={"ID":"9989c28e-0ee9-4684-a305-152f684d7a5d","Type":"ContainerStarted","Data":"d7edfa8df778d9be4c4be7bfe1fb1f4503b1826686f95dec2d02fe25929cd2d4"} Jul 08 19:05:14.387901 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.387834 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" event={"ID":"9989c28e-0ee9-4684-a305-152f684d7a5d","Type":"ContainerStarted","Data":"6a25dfee7e1b4e751c171f23abf42c84647b17f5fc97333e73a9fafbeeb6fa68"} Jul 08 19:05:14.387901 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.387861 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" event={"ID":"9989c28e-0ee9-4684-a305-152f684d7a5d","Type":"ContainerStarted","Data":"84b7b83dc8277157cb4408657fdc233e205b05422013f48b9f8cf8dd722c6a9b"} Jul 08 19:05:14.585431 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.585384 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.588507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.588471 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.741621 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.741589 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99"] Jul 08 19:05:14.744366 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.744341 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.747716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.747141 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 19:05:14.747716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.747184 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 19:05:14.747716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.747401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 19:05:14.747716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.747577 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-86ccx\"" Jul 08 19:05:14.747716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.747616 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-36motceiofhgd\"" Jul 08 19:05:14.747716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.747713 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 19:05:14.748066 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.747812 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 19:05:14.758461 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.757219 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99"] Jul 08 19:05:14.790229 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790116 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-tls\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.790229 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790152 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.790229 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpflb\" (UniqueName: \"kubernetes.io/projected/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-kube-api-access-vpflb\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.790478 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790250 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-metrics-client-ca\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.790478 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790285 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-grpc-tls\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.790478 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.790478 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.790478 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.790398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.803046 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.803001 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:05:14.891236 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-metrics-client-ca\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891407 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-grpc-tls\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891407 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891276 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891407 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891330 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891407 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891648 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891416 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-tls\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891648 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891648 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpflb\" (UniqueName: \"kubernetes.io/projected/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-kube-api-access-vpflb\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.891988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.891970 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-metrics-client-ca\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.893706 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.893677 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 19:05:14.893880 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.893856 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 19:05:14.893986 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.893857 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 19:05:14.894062 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.894039 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 19:05:14.894205 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.894186 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-36motceiofhgd\"" Jul 08 19:05:14.894318 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.894295 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 19:05:14.901676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.901367 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpflb\" (UniqueName: \"kubernetes.io/projected/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-kube-api-access-vpflb\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.904329 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.904271 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-grpc-tls\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.904329 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.904273 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.904785 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.904763 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.905575 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.905531 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.905740 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.905719 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-tls\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:14.910475 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:14.910436 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9031726c-a343-4f74-8d8d-ec4cfa5e9ec2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77d5b8dbf5-c9k99\" (UID: \"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2\") " pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:15.061225 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:15.061146 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-86ccx\"" Jul 08 19:05:15.068946 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:15.068743 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:15.399049 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:15.399004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" event={"ID":"9989c28e-0ee9-4684-a305-152f684d7a5d","Type":"ContainerStarted","Data":"363e2fbdeb744f71c97f6cd45d3dd24e97f6dd4d98c355c4e5c00ac65c1cf246"} Jul 08 19:05:15.410587 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:15.409144 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tr92" event={"ID":"0e30e11f-9fec-4d54-8f79-01da25f8b3ae","Type":"ContainerStarted","Data":"56256e2e9cf55081eb36251d2c99ebd27a0f1f3644f0e6d501ab269ff4307657"} Jul 08 19:05:15.432281 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:15.432216 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-24sdz" podStartSLOduration=2.114619852 podStartE2EDuration="3.432192799s" totalImagesPullingTime="1.317572947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:15.429857681 +0000 UTC m=+60.869160316" watchObservedRunningTime="2026-07-08 19:05:15.432192799 +0000 UTC m=+60.871495430" Jul 08 19:05:15.469252 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:15.469225 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:05:15.479311 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:15.479287 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99"] Jul 08 19:05:15.763129 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:15.763084 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9031726c_a343_4f74_8d8d_ec4cfa5e9ec2.slice/crio-eeb1131b91ee233a40b8603894f7a0dc6975e4c9a80018fe30a32987136218a9 WatchSource:0}: Error finding container eeb1131b91ee233a40b8603894f7a0dc6975e4c9a80018fe30a32987136218a9: Status 404 returned error can't find the container with id eeb1131b91ee233a40b8603894f7a0dc6975e4c9a80018fe30a32987136218a9 Jul 08 19:05:15.764249 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:15.764220 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f99285_4558_48e6_b72d_a237e32d368e.slice/crio-2c3761a5ff487db3ea387d5263be9ba9db95e9f52cb73753bdda366067d43666 WatchSource:0}: Error finding container 2c3761a5ff487db3ea387d5263be9ba9db95e9f52cb73753bdda366067d43666: Status 404 returned error can't find the container with id 2c3761a5ff487db3ea387d5263be9ba9db95e9f52cb73753bdda366067d43666 Jul 08 19:05:16.416466 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.416425 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" event={"ID":"e3d39085-2b8b-4ca0-a91c-93912b9a847a","Type":"ContainerStarted","Data":"0498a87995c05f58098f35d1c0a91d4b32134ce4704fe1b4ccbc3004d7e885ea"} Jul 08 19:05:16.416937 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.416507 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" event={"ID":"e3d39085-2b8b-4ca0-a91c-93912b9a847a","Type":"ContainerStarted","Data":"b81e49e48cfc6dce564501d4d40b3104b1deca1d09a8d3390bfb4e7a84322d1f"} Jul 08 19:05:16.416937 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.416561 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" event={"ID":"e3d39085-2b8b-4ca0-a91c-93912b9a847a","Type":"ContainerStarted","Data":"97dae2dbb3a3301722fe36db34c13a9eb0fcb2f40430db10c75e40e01b3fec63"} Jul 08 19:05:16.417897 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.417868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" event={"ID":"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2","Type":"ContainerStarted","Data":"eeb1131b91ee233a40b8603894f7a0dc6975e4c9a80018fe30a32987136218a9"} Jul 08 19:05:16.419671 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.419646 2569 generic.go:356] "Generic (PLEG): container finished" podID="0e30e11f-9fec-4d54-8f79-01da25f8b3ae" containerID="56256e2e9cf55081eb36251d2c99ebd27a0f1f3644f0e6d501ab269ff4307657" exitCode=0 Jul 08 19:05:16.419811 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.419709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tr92" event={"ID":"0e30e11f-9fec-4d54-8f79-01da25f8b3ae","Type":"ContainerDied","Data":"56256e2e9cf55081eb36251d2c99ebd27a0f1f3644f0e6d501ab269ff4307657"} Jul 08 19:05:16.421357 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.421336 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerStarted","Data":"2c3761a5ff487db3ea387d5263be9ba9db95e9f52cb73753bdda366067d43666"} Jul 08 19:05:16.436089 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:16.436050 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-sv47r" podStartSLOduration=2.501919388 podStartE2EDuration="4.436035648s" totalImagesPullingTime="1.93411626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:16.434646024 +0000 UTC m=+61.873948680" watchObservedRunningTime="2026-07-08 19:05:16.436035648 +0000 UTC m=+61.875338257" Jul 08 19:05:17.078119 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.078082 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84c589b7bd-kwt7k"] Jul 08 19:05:17.080703 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.080678 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.084100 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.084068 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9o2ivbnlq87dc\"" Jul 08 19:05:17.084192 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.084105 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 08 19:05:17.084253 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.084106 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 19:05:17.084253 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.084197 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 08 19:05:17.084362 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.084329 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zd4q6\"" Jul 08 19:05:17.084362 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.084348 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 08 19:05:17.100105 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.100081 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84c589b7bd-kwt7k"] Jul 08 19:05:17.112258 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.112231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-secret-metrics-server-client-certs\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.112396 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.112272 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a445f661-6c1f-44d8-882b-61b6e9411392-metrics-server-audit-profiles\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.112396 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.112308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a445f661-6c1f-44d8-882b-61b6e9411392-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.112396 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.112336 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-client-ca-bundle\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.112396 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.112380 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-secret-metrics-server-tls\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.112629 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.112402 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a445f661-6c1f-44d8-882b-61b6e9411392-audit-log\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.112629 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.112449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tz4\" (UniqueName: \"kubernetes.io/projected/a445f661-6c1f-44d8-882b-61b6e9411392-kube-api-access-76tz4\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.213456 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.213413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tz4\" (UniqueName: \"kubernetes.io/projected/a445f661-6c1f-44d8-882b-61b6e9411392-kube-api-access-76tz4\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.213651 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.213518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-secret-metrics-server-client-certs\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.213651 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.213561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a445f661-6c1f-44d8-882b-61b6e9411392-metrics-server-audit-profiles\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.213806 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.213758 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a445f661-6c1f-44d8-882b-61b6e9411392-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.213806 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.213796 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-client-ca-bundle\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.213899 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.213867 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-secret-metrics-server-tls\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.213951 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.213919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a445f661-6c1f-44d8-882b-61b6e9411392-audit-log\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.214485 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.214460 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a445f661-6c1f-44d8-882b-61b6e9411392-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.214691 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.214668 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a445f661-6c1f-44d8-882b-61b6e9411392-metrics-server-audit-profiles\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.215197 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.215173 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a445f661-6c1f-44d8-882b-61b6e9411392-audit-log\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.217881 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.217368 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-secret-metrics-server-tls\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.217881 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.217832 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-client-ca-bundle\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.219810 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.219780 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a445f661-6c1f-44d8-882b-61b6e9411392-secret-metrics-server-client-certs\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.223047 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.223015 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tz4\" (UniqueName: \"kubernetes.io/projected/a445f661-6c1f-44d8-882b-61b6e9411392-kube-api-access-76tz4\") pod \"metrics-server-84c589b7bd-kwt7k\" (UID: \"a445f661-6c1f-44d8-882b-61b6e9411392\") " pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.293292 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.293266 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbj8q" Jul 08 19:05:17.392836 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.392756 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:17.427602 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.427565 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tr92" event={"ID":"0e30e11f-9fec-4d54-8f79-01da25f8b3ae","Type":"ContainerStarted","Data":"d036e8e2cfb64b4ef78904055d72f7a00d9b2410e366b2c85ec3b0ae7f949082"} Jul 08 19:05:17.428005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.427668 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tr92" event={"ID":"0e30e11f-9fec-4d54-8f79-01da25f8b3ae","Type":"ContainerStarted","Data":"4556ac4affeb259a519642a569ce260c25f951cccbf200abd1cf7c6d4c2f7d2a"} Jul 08 19:05:17.622164 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.622110 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9tr92" podStartSLOduration=3.958200775 podStartE2EDuration="5.622091073s" totalImagesPullingTime="1.567551316s" totalInitContainerRuntime="96.338982ms" isStatefulPod=true podCreationTimestamp="2026-07-08 19:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:17.469735958 +0000 UTC m=+62.909038590" watchObservedRunningTime="2026-07-08 19:05:17.622091073 +0000 UTC m=+63.061393703" Jul 08 19:05:17.622400 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.622380 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99"] Jul 08 19:05:17.624497 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.624476 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:17.649268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.642844 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-59l4x\"" Jul 08 19:05:17.649268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.643124 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 08 19:05:17.668753 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.668728 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99"] Jul 08 19:05:17.718184 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.717996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8phs\" (UniqueName: \"kubernetes.io/projected/65154e7f-df89-4368-95ff-2b4147be5f9b-kube-api-access-n8phs\") pod \"monitoring-plugin-d5dcff7cf-x4g99\" (UID: \"65154e7f-df89-4368-95ff-2b4147be5f9b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:17.718184 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.718090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65154e7f-df89-4368-95ff-2b4147be5f9b-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-x4g99\" (UID: \"65154e7f-df89-4368-95ff-2b4147be5f9b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:17.765349 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.765324 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dc848fbdb-l9gd9"] Jul 08 19:05:17.767307 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.767287 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.778825 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.778798 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 19:05:17.778990 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.778904 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 19:05:17.779270 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.779245 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 19:05:17.779341 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.779285 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 19:05:17.779575 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.779532 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 19:05:17.779636 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.779618 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tk955\"" Jul 08 19:05:17.791505 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.791475 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc848fbdb-l9gd9"] Jul 08 19:05:17.807569 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.807522 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84c589b7bd-kwt7k"] Jul 08 19:05:17.819045 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-serving-cert\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.819180 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819064 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65154e7f-df89-4368-95ff-2b4147be5f9b-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-x4g99\" (UID: \"65154e7f-df89-4368-95ff-2b4147be5f9b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:17.819180 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-oauth-serving-cert\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.819261 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819196 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-console-config\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.819261 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819230 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8phs\" (UniqueName: \"kubernetes.io/projected/65154e7f-df89-4368-95ff-2b4147be5f9b-kube-api-access-n8phs\") pod \"monitoring-plugin-d5dcff7cf-x4g99\" (UID: \"65154e7f-df89-4368-95ff-2b4147be5f9b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:17.819333 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-oauth-config\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.819333 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819303 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-service-ca\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.819435 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.819349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqwl\" (UniqueName: \"kubernetes.io/projected/97092863-40a5-4bc4-822c-22c47d6f1d96-kube-api-access-5sqwl\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.822265 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.822217 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/65154e7f-df89-4368-95ff-2b4147be5f9b-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-x4g99\" (UID: \"65154e7f-df89-4368-95ff-2b4147be5f9b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:17.833723 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.833689 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8phs\" (UniqueName: \"kubernetes.io/projected/65154e7f-df89-4368-95ff-2b4147be5f9b-kube-api-access-n8phs\") pod \"monitoring-plugin-d5dcff7cf-x4g99\" (UID: \"65154e7f-df89-4368-95ff-2b4147be5f9b\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:17.920261 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.920223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-serving-cert\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.920451 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.920318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-oauth-serving-cert\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.920451 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.920346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-console-config\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.920451 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.920389 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-oauth-config\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.920451 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.920410 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-service-ca\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.920741 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.920488 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqwl\" (UniqueName: \"kubernetes.io/projected/97092863-40a5-4bc4-822c-22c47d6f1d96-kube-api-access-5sqwl\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.921708 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.921652 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-oauth-serving-cert\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.921708 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.921662 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-console-config\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.921861 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.921732 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-service-ca\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.923504 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.923480 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-oauth-config\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.923703 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.923506 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-serving-cert\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.931307 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.931269 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqwl\" (UniqueName: \"kubernetes.io/projected/97092863-40a5-4bc4-822c-22c47d6f1d96-kube-api-access-5sqwl\") pod \"console-6dc848fbdb-l9gd9\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:17.954889 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:17.954845 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:18.078143 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:18.077854 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:18.319153 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:18.319069 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda445f661_6c1f_44d8_882b_61b6e9411392.slice/crio-7a10f23d9b40739d861b5d16c2d6b60ed5926c10c7502820bfeaf62d04d0b8bb WatchSource:0}: Error finding container 7a10f23d9b40739d861b5d16c2d6b60ed5926c10c7502820bfeaf62d04d0b8bb: Status 404 returned error can't find the container with id 7a10f23d9b40739d861b5d16c2d6b60ed5926c10c7502820bfeaf62d04d0b8bb Jul 08 19:05:18.360375 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:18.360350 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w8k79" Jul 08 19:05:18.438619 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:18.438587 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" event={"ID":"a445f661-6c1f-44d8-882b-61b6e9411392","Type":"ContainerStarted","Data":"7a10f23d9b40739d861b5d16c2d6b60ed5926c10c7502820bfeaf62d04d0b8bb"} Jul 08 19:05:18.441023 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:18.440898 2569 generic.go:356] "Generic (PLEG): container finished" podID="84f99285-4558-48e6-b72d-a237e32d368e" containerID="4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4" exitCode=0 Jul 08 19:05:18.441120 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:18.441036 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4"} Jul 08 19:05:18.495516 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:18.495465 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc848fbdb-l9gd9"] Jul 08 19:05:18.510586 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:18.510556 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97092863_40a5_4bc4_822c_22c47d6f1d96.slice/crio-539eaf8dddcff2508936501ee0d794e3e285372e477b4036abf2c10ad668c776 WatchSource:0}: Error finding container 539eaf8dddcff2508936501ee0d794e3e285372e477b4036abf2c10ad668c776: Status 404 returned error can't find the container with id 539eaf8dddcff2508936501ee0d794e3e285372e477b4036abf2c10ad668c776 Jul 08 19:05:18.520327 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:18.520305 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99"] Jul 08 19:05:18.522720 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:18.522695 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65154e7f_df89_4368_95ff_2b4147be5f9b.slice/crio-5e38449210f434b3318634c30fb35e016c9fd057c5d97ec7cfd6416bfe6ffbb0 WatchSource:0}: Error finding container 5e38449210f434b3318634c30fb35e016c9fd057c5d97ec7cfd6416bfe6ffbb0: Status 404 returned error can't find the container with id 5e38449210f434b3318634c30fb35e016c9fd057c5d97ec7cfd6416bfe6ffbb0 Jul 08 19:05:19.017941 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.017902 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:05:19.024839 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.024801 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.029880 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.029852 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 19:05:19.029880 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.029864 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3rn0nhuf4d360\"" Jul 08 19:05:19.030275 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.029955 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 19:05:19.030275 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030049 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 19:05:19.030275 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030272 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-px2wp\"" Jul 08 19:05:19.030502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030333 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 19:05:19.030502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030381 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 19:05:19.030502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030445 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 19:05:19.030743 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030336 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 19:05:19.030743 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030658 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 19:05:19.030743 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030680 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 19:05:19.030743 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030720 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 19:05:19.030999 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.030920 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 19:05:19.031577 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.031131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 19:05:19.031577 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.031333 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 19:05:19.036743 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.036723 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 19:05:19.053517 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.052522 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:05:19.135661 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.135838 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.135838 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.135838 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135768 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.135838 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.135838 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135878 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.135969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv5f4\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-kube-api-access-dv5f4\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136378 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136378 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136167 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136378 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-config\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136378 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136378 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136378 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.136658 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.136406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237410 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237609 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237609 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv5f4\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-kube-api-access-dv5f4\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237609 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237496 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237609 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237609 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-config\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237646 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237740 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237763 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.237886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.238356 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.238356 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237942 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.238356 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.237977 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.238356 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.238001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.240903 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.240282 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.240903 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.240572 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.242002 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.241935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.243571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.243519 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.243982 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.243995 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.244172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.244281 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.244739 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.245218 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.245570 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.245774 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.245942 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-config\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.246245 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.246208 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.247297 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.247265 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.247963 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.247899 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.248329 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.248310 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.248875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.248850 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.249775 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.249757 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.253370 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.253346 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv5f4\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-kube-api-access-dv5f4\") pod \"prometheus-k8s-0\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.337998 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.337916 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:19.472267 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.472230 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc848fbdb-l9gd9" event={"ID":"97092863-40a5-4bc4-822c-22c47d6f1d96","Type":"ContainerStarted","Data":"539eaf8dddcff2508936501ee0d794e3e285372e477b4036abf2c10ad668c776"} Jul 08 19:05:19.475199 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.475167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" event={"ID":"65154e7f-df89-4368-95ff-2b4147be5f9b","Type":"ContainerStarted","Data":"5e38449210f434b3318634c30fb35e016c9fd057c5d97ec7cfd6416bfe6ffbb0"} Jul 08 19:05:19.483455 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.483264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" event={"ID":"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2","Type":"ContainerStarted","Data":"d52addc9e772618210de44f9a3f224cd0d8e560c6146ddfaa69af9cf1cc1bbbc"} Jul 08 19:05:19.483455 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.483350 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" event={"ID":"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2","Type":"ContainerStarted","Data":"1afae301546ab342ed1f072bd2bae8274eb18bfa69cfb14d7c7300a7989df8f7"} Jul 08 19:05:19.483455 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.483390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" event={"ID":"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2","Type":"ContainerStarted","Data":"f51ce81471400cc4d7d6de0d2137f7b7f5c9473e09906cc330d7ade075ec0c60"} Jul 08 19:05:19.583847 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:19.583500 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:05:20.491581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.491524 2569 generic.go:356] "Generic (PLEG): container finished" podID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" exitCode=0 Jul 08 19:05:20.491943 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.491685 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd"} Jul 08 19:05:20.491943 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.491753 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerStarted","Data":"770fbc8ff294ff5dcc98b1fe87ef8865d5aa052d19bd09fd40789aa907569187"} Jul 08 19:05:20.498248 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.498162 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" event={"ID":"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2","Type":"ContainerStarted","Data":"dd4bc4b00f2c11139ccca230c018f544a94146bd76a90191270235ca416e2de8"} Jul 08 19:05:20.765319 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.765139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:05:20.767529 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.767507 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:05:20.780909 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.780875 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1ce6677-e64a-4c21-9faf-a715c7a83d5f-original-pull-secret\") pod \"global-pull-secret-syncer-wc6c2\" (UID: \"b1ce6677-e64a-4c21-9faf-a715c7a83d5f\") " pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:05:20.866568 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.866299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:05:20.866568 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.866479 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:05:20.868966 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.868810 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:05:20.868966 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.868827 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:05:20.879039 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.879004 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:05:20.879474 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.879454 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d613ce2-7191-434e-8b6d-c8960eb95403-metrics-certs\") pod \"network-metrics-daemon-cdtvk\" (UID: \"0d613ce2-7191-434e-8b6d-c8960eb95403\") " pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:05:20.890930 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.890879 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpkq\" (UniqueName: \"kubernetes.io/projected/4025f320-f92c-42a0-aa12-69e475af26b9-kube-api-access-znpkq\") pod \"network-check-target-t9zpq\" (UID: \"4025f320-f92c-42a0-aa12-69e475af26b9\") " pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:05:20.905268 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.905241 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kpjrj\"" Jul 08 19:05:20.913694 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:20.913672 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cdtvk" Jul 08 19:05:21.010423 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:21.010219 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc6c2" Jul 08 19:05:21.023026 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:21.022965 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gzqsf\"" Jul 08 19:05:21.031137 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:21.031103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:05:21.506206 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:21.506169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" event={"ID":"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2","Type":"ContainerStarted","Data":"90275524a51f7eeade4c82d349f2c0990482a56446935ea006e1f8d80bbd0450"} Jul 08 19:05:22.993155 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:22.992883 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t9zpq"] Jul 08 19:05:23.012307 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.012284 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wc6c2"] Jul 08 19:05:23.030529 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.030496 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cdtvk"] Jul 08 19:05:23.045572 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:23.045500 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4025f320_f92c_42a0_aa12_69e475af26b9.slice/crio-7ccd04fbc32e0a2f3ae0b4e807fb1ddb209fd79f3ea307a12cbb4900eb0814cd WatchSource:0}: Error finding container 7ccd04fbc32e0a2f3ae0b4e807fb1ddb209fd79f3ea307a12cbb4900eb0814cd: Status 404 returned error can't find the container with id 7ccd04fbc32e0a2f3ae0b4e807fb1ddb209fd79f3ea307a12cbb4900eb0814cd Jul 08 19:05:23.046164 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:23.046128 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ce6677_e64a_4c21_9faf_a715c7a83d5f.slice/crio-5660d35b7abf17c8c8da08b657795f67f77b95e8c0e7fe1c92822e87bd451a72 WatchSource:0}: Error finding container 5660d35b7abf17c8c8da08b657795f67f77b95e8c0e7fe1c92822e87bd451a72: Status 404 returned error can't find the container with id 5660d35b7abf17c8c8da08b657795f67f77b95e8c0e7fe1c92822e87bd451a72 Jul 08 19:05:23.047439 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:05:23.047418 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d613ce2_7191_434e_8b6d_c8960eb95403.slice/crio-a040b12418d4cca87f9205d114e770873aca18e93156cbf3928aa27dbab534b3 WatchSource:0}: Error finding container a040b12418d4cca87f9205d114e770873aca18e93156cbf3928aa27dbab534b3: Status 404 returned error can't find the container with id a040b12418d4cca87f9205d114e770873aca18e93156cbf3928aa27dbab534b3 Jul 08 19:05:23.521913 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.521709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerStarted","Data":"5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad"} Jul 08 19:05:23.521913 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.521799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerStarted","Data":"db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0"} Jul 08 19:05:23.521913 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.521845 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerStarted","Data":"8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47"} Jul 08 19:05:23.526605 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.525661 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" event={"ID":"9031726c-a343-4f74-8d8d-ec4cfa5e9ec2","Type":"ContainerStarted","Data":"b6303a482a4c2b9d56f2b8c86c9f9b8f635f7a9742ee90a6ea31778f2d548203"} Jul 08 19:05:23.526716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.526612 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:23.530475 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.529288 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" event={"ID":"a445f661-6c1f-44d8-882b-61b6e9411392","Type":"ContainerStarted","Data":"d17510fa3dce099004106cf9489ad89e848de1e656e497c48b514f96f50de2c2"} Jul 08 19:05:23.532847 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.532828 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" Jul 08 19:05:23.533530 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.533480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cdtvk" event={"ID":"0d613ce2-7191-434e-8b6d-c8960eb95403","Type":"ContainerStarted","Data":"a040b12418d4cca87f9205d114e770873aca18e93156cbf3928aa27dbab534b3"} Jul 08 19:05:23.534850 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.534795 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t9zpq" event={"ID":"4025f320-f92c-42a0-aa12-69e475af26b9","Type":"ContainerStarted","Data":"7ccd04fbc32e0a2f3ae0b4e807fb1ddb209fd79f3ea307a12cbb4900eb0814cd"} Jul 08 19:05:23.536670 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.536643 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc848fbdb-l9gd9" event={"ID":"97092863-40a5-4bc4-822c-22c47d6f1d96","Type":"ContainerStarted","Data":"1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2"} Jul 08 19:05:23.547566 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.545701 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" event={"ID":"65154e7f-df89-4368-95ff-2b4147be5f9b","Type":"ContainerStarted","Data":"a5f00e3fb8d3539dc9729405add345e486d2596c624da08ce92b10695858e6c4"} Jul 08 19:05:23.547566 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.546201 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:23.551495 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.551409 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77d5b8dbf5-c9k99" podStartSLOduration=5.56769454 podStartE2EDuration="9.551390278s" totalImagesPullingTime="3.983695738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:23.550593183 +0000 UTC m=+68.989895836" watchObservedRunningTime="2026-07-08 19:05:23.551390278 +0000 UTC m=+68.990692910" Jul 08 19:05:23.552085 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.552053 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wc6c2" event={"ID":"b1ce6677-e64a-4c21-9faf-a715c7a83d5f","Type":"ContainerStarted","Data":"5660d35b7abf17c8c8da08b657795f67f77b95e8c0e7fe1c92822e87bd451a72"} Jul 08 19:05:23.552702 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.552679 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" Jul 08 19:05:23.596111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.593985 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dc848fbdb-l9gd9" podStartSLOduration=2.019055359 podStartE2EDuration="6.593968219s" totalImagesPullingTime="4.57491286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:23.593906237 +0000 UTC m=+69.033208891" watchObservedRunningTime="2026-07-08 19:05:23.593968219 +0000 UTC m=+69.033270853" Jul 08 19:05:23.596111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.594930 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-x4g99" podStartSLOduration=2.304314941 podStartE2EDuration="6.594918027s" totalImagesPullingTime="4.290603086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:23.570424984 +0000 UTC m=+69.009727620" watchObservedRunningTime="2026-07-08 19:05:23.594918027 +0000 UTC m=+69.034220658" Jul 08 19:05:23.614814 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:23.613875 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" podStartSLOduration=2.131768163 podStartE2EDuration="6.613853464s" totalImagesPullingTime="4.482085301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:23.61271167 +0000 UTC m=+69.052014328" watchObservedRunningTime="2026-07-08 19:05:23.613853464 +0000 UTC m=+69.053156096" Jul 08 19:05:24.568774 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:24.566857 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerStarted","Data":"f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495"} Jul 08 19:05:24.568774 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:24.566941 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerStarted","Data":"e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194"} Jul 08 19:05:24.568774 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:24.566987 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerStarted","Data":"16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f"} Jul 08 19:05:24.599588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:24.598768 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.302655923 podStartE2EDuration="11.598743808s" totalImagesPullingTime="6.277815858s" totalInitContainerRuntime="18.272027ms" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:24.596902074 +0000 UTC m=+70.036204727" watchObservedRunningTime="2026-07-08 19:05:24.598743808 +0000 UTC m=+70.038046441" Jul 08 19:05:25.575519 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:25.575482 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cdtvk" event={"ID":"0d613ce2-7191-434e-8b6d-c8960eb95403","Type":"ContainerStarted","Data":"b80b4ad50167e758ecaed6e96487989655abef6aff4955378c8f2cc53aea2b37"} Jul 08 19:05:25.575945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:25.575556 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cdtvk" event={"ID":"0d613ce2-7191-434e-8b6d-c8960eb95403","Type":"ContainerStarted","Data":"8a96f4ff9abd59cf840ad2e0ff7a9d9aff3c479ae6a723da52f5499b22be4e19"} Jul 08 19:05:25.600631 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:25.600118 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cdtvk" podStartSLOduration=68.931412941 podStartE2EDuration="1m10.600095134s" totalImagesPullingTime="1.668682193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:25.597974244 +0000 UTC m=+71.037276901" watchObservedRunningTime="2026-07-08 19:05:25.600095134 +0000 UTC m=+71.039397768" Jul 08 19:05:27.587023 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:27.586989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerStarted","Data":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} Jul 08 19:05:27.587457 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:27.587076 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerStarted","Data":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} Jul 08 19:05:28.078465 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.078433 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:28.078611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.078573 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:28.089668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.089641 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:28.595996 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.595883 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerStarted","Data":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} Jul 08 19:05:28.595996 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.595961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerStarted","Data":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} Jul 08 19:05:28.596449 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.596003 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerStarted","Data":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} Jul 08 19:05:28.596449 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.596040 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerStarted","Data":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} Jul 08 19:05:28.597653 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.597613 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t9zpq" event={"ID":"4025f320-f92c-42a0-aa12-69e475af26b9","Type":"ContainerStarted","Data":"deb07f9ceb550170e5003b1fd4efe9062951605446d1e974d0506bb1716088ed"} Jul 08 19:05:28.601947 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.601922 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:28.640776 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.640720 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.448821282 podStartE2EDuration="10.640700325s" totalImagesPullingTime="6.162644925s" totalInitContainerRuntime="29.234118ms" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:28.639369726 +0000 UTC m=+74.078672379" watchObservedRunningTime="2026-07-08 19:05:28.640700325 +0000 UTC m=+74.080002957" Jul 08 19:05:28.682771 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:28.682721 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t9zpq" podStartSLOduration=68.731417095 podStartE2EDuration="1m13.682702593s" totalImagesPullingTime="4.951285498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:28.68089525 +0000 UTC m=+74.120197909" watchObservedRunningTime="2026-07-08 19:05:28.682702593 +0000 UTC m=+74.122005225" Jul 08 19:05:29.338781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:29.338739 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:05:29.602726 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:29.602645 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:05:30.850724 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:30.850694 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc848fbdb-l9gd9"] Jul 08 19:05:37.393933 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:37.393782 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:37.393933 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:37.393863 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:38.646638 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:38.646591 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wc6c2" event={"ID":"b1ce6677-e64a-4c21-9faf-a715c7a83d5f","Type":"ContainerStarted","Data":"75dd1aeb08de067bd7a22540be7df9f2c0c955e03e8ead2fa8e796dd78ea816b"} Jul 08 19:05:38.648658 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:38.648622 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-vw4jc" event={"ID":"164eba15-890e-4e1b-a0e6-e7750d37db3d","Type":"ContainerStarted","Data":"4bd1614a6ade95da1f4952aae62717c7c3003499c13f353ab4d393b1afcbb30a"} Jul 08 19:05:38.648805 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:38.648683 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-vw4jc" Jul 08 19:05:38.666430 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:38.666371 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wc6c2" podStartSLOduration=69.202566987 podStartE2EDuration="1m23.666348788s" totalImagesPullingTime="14.463781801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:38.664947961 +0000 UTC m=+84.104250613" watchObservedRunningTime="2026-07-08 19:05:38.666348788 +0000 UTC m=+84.105651420" Jul 08 19:05:38.667499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:38.667474 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-vw4jc" Jul 08 19:05:38.684577 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:38.684513 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-vw4jc" podStartSLOduration=2.683752271 podStartE2EDuration="31.684495571s" totalImagesPullingTime="29.0007433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:38.682491664 +0000 UTC m=+84.121794324" watchObservedRunningTime="2026-07-08 19:05:38.684495571 +0000 UTC m=+84.123798202" Jul 08 19:05:55.880348 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:55.880288 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6dc848fbdb-l9gd9" podUID="97092863-40a5-4bc4-822c-22c47d6f1d96" containerName="console" containerID="cri-o://1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2" gracePeriod=15 Jul 08 19:05:56.158352 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.158323 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:56.181912 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.181882 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-oauth-config\") pod \"97092863-40a5-4bc4-822c-22c47d6f1d96\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " Jul 08 19:05:56.182042 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.181923 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-oauth-serving-cert\") pod \"97092863-40a5-4bc4-822c-22c47d6f1d96\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " Jul 08 19:05:56.182042 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.181956 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-service-ca\") pod \"97092863-40a5-4bc4-822c-22c47d6f1d96\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " Jul 08 19:05:56.182042 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.182005 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-serving-cert\") pod \"97092863-40a5-4bc4-822c-22c47d6f1d96\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " Jul 08 19:05:56.182042 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.182038 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sqwl\" (UniqueName: \"kubernetes.io/projected/97092863-40a5-4bc4-822c-22c47d6f1d96-kube-api-access-5sqwl\") pod \"97092863-40a5-4bc4-822c-22c47d6f1d96\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " Jul 08 19:05:56.182241 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.182078 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-console-config\") pod \"97092863-40a5-4bc4-822c-22c47d6f1d96\" (UID: \"97092863-40a5-4bc4-822c-22c47d6f1d96\") " Jul 08 19:05:56.182465 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.182437 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97092863-40a5-4bc4-822c-22c47d6f1d96" (UID: "97092863-40a5-4bc4-822c-22c47d6f1d96"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:05:56.182586 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.182368 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-service-ca" (OuterVolumeSpecName: "service-ca") pod "97092863-40a5-4bc4-822c-22c47d6f1d96" (UID: "97092863-40a5-4bc4-822c-22c47d6f1d96"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:05:56.182586 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.182512 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-console-config" (OuterVolumeSpecName: "console-config") pod "97092863-40a5-4bc4-822c-22c47d6f1d96" (UID: "97092863-40a5-4bc4-822c-22c47d6f1d96"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:05:56.184302 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.184284 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97092863-40a5-4bc4-822c-22c47d6f1d96" (UID: "97092863-40a5-4bc4-822c-22c47d6f1d96"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:05:56.184747 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.184721 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97092863-40a5-4bc4-822c-22c47d6f1d96-kube-api-access-5sqwl" (OuterVolumeSpecName: "kube-api-access-5sqwl") pod "97092863-40a5-4bc4-822c-22c47d6f1d96" (UID: "97092863-40a5-4bc4-822c-22c47d6f1d96"). InnerVolumeSpecName "kube-api-access-5sqwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:05:56.184816 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.184734 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97092863-40a5-4bc4-822c-22c47d6f1d96" (UID: "97092863-40a5-4bc4-822c-22c47d6f1d96"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:05:56.282686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.282652 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-oauth-config\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:05:56.282686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.282680 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-oauth-serving-cert\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:05:56.282686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.282690 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-service-ca\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:05:56.282686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.282698 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97092863-40a5-4bc4-822c-22c47d6f1d96-console-serving-cert\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:05:56.282955 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.282706 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sqwl\" (UniqueName: \"kubernetes.io/projected/97092863-40a5-4bc4-822c-22c47d6f1d96-kube-api-access-5sqwl\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:05:56.282955 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.282727 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97092863-40a5-4bc4-822c-22c47d6f1d96-console-config\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:05:56.732288 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.732250 2569 generic.go:356] "Generic (PLEG): container finished" podID="97092863-40a5-4bc4-822c-22c47d6f1d96" containerID="1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2" exitCode=0 Jul 08 19:05:56.732441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.732321 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc848fbdb-l9gd9" Jul 08 19:05:56.732441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.732327 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc848fbdb-l9gd9" event={"ID":"97092863-40a5-4bc4-822c-22c47d6f1d96","Type":"ContainerDied","Data":"1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2"} Jul 08 19:05:56.732441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.732400 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc848fbdb-l9gd9" event={"ID":"97092863-40a5-4bc4-822c-22c47d6f1d96","Type":"ContainerDied","Data":"539eaf8dddcff2508936501ee0d794e3e285372e477b4036abf2c10ad668c776"} Jul 08 19:05:56.732441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.732420 2569 scope.go:117] "RemoveContainer" containerID="1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2" Jul 08 19:05:56.741189 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.741168 2569 scope.go:117] "RemoveContainer" containerID="1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2" Jul 08 19:05:56.741461 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:05:56.741442 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2\": container with ID starting with 1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2 not found: ID does not exist" containerID="1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2" Jul 08 19:05:56.741520 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.741469 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2"} err="failed to get container status \"1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2\": rpc error: code = NotFound desc = could not find container \"1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2\": container with ID starting with 1f22c529d90079381fcb2207cad36024a2921af9547f706a0f15aac7df5b70f2 not found: ID does not exist" Jul 08 19:05:56.755249 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.755222 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc848fbdb-l9gd9"] Jul 08 19:05:56.759085 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:56.759063 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dc848fbdb-l9gd9"] Jul 08 19:05:57.098690 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:57.098604 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97092863-40a5-4bc4-822c-22c47d6f1d96" path="/var/lib/kubelet/pods/97092863-40a5-4bc4-822c-22c47d6f1d96/volumes" Jul 08 19:05:57.398608 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:57.398522 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:05:57.402317 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:05:57.402291 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84c589b7bd-kwt7k" Jul 08 19:06:00.609773 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:00.609741 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t9zpq" Jul 08 19:06:19.339321 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:19.339284 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:19.395581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:19.395533 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:19.838652 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:19.838624 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:33.112770 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.112735 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:06:33.113247 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.112974 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="alertmanager" containerID="cri-o://8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47" gracePeriod=120 Jul 08 19:06:33.113247 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.112986 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy" containerID="cri-o://16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f" gracePeriod=120 Jul 08 19:06:33.113247 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.113037 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-metric" containerID="cri-o://e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194" gracePeriod=120 Jul 08 19:06:33.113988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.113699 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-web" containerID="cri-o://5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad" gracePeriod=120 Jul 08 19:06:33.114687 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.114151 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="prom-label-proxy" containerID="cri-o://f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495" gracePeriod=120 Jul 08 19:06:33.119107 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.114916 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="config-reloader" containerID="cri-o://db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0" gracePeriod=120 Jul 08 19:06:33.880007 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.879976 2569 generic.go:356] "Generic (PLEG): container finished" podID="84f99285-4558-48e6-b72d-a237e32d368e" containerID="f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495" exitCode=0 Jul 08 19:06:33.880007 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.879998 2569 generic.go:356] "Generic (PLEG): container finished" podID="84f99285-4558-48e6-b72d-a237e32d368e" containerID="16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f" exitCode=0 Jul 08 19:06:33.880007 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880005 2569 generic.go:356] "Generic (PLEG): container finished" podID="84f99285-4558-48e6-b72d-a237e32d368e" containerID="5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad" exitCode=0 Jul 08 19:06:33.880007 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880010 2569 generic.go:356] "Generic (PLEG): container finished" podID="84f99285-4558-48e6-b72d-a237e32d368e" containerID="db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0" exitCode=0 Jul 08 19:06:33.880007 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880015 2569 generic.go:356] "Generic (PLEG): container finished" podID="84f99285-4558-48e6-b72d-a237e32d368e" containerID="8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47" exitCode=0 Jul 08 19:06:33.880285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880051 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495"} Jul 08 19:06:33.880285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880136 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f"} Jul 08 19:06:33.880285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880162 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad"} Jul 08 19:06:33.880285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880187 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0"} Jul 08 19:06:33.880285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:33.880210 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47"} Jul 08 19:06:34.367486 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.367462 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:34.533832 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.533734 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-web\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.533832 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.533776 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.533832 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.533802 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534105 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.533997 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-config-volume\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534105 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534034 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-trusted-ca-bundle\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534105 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534080 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-main-db\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534254 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534116 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-cluster-tls-config\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534254 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534149 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534254 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534178 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-metrics-client-ca\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534254 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534204 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-tls-assets\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534429 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534273 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jn67\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-kube-api-access-9jn67\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534429 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534313 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-config-out\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534429 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534339 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-web-config\") pod \"84f99285-4558-48e6-b72d-a237e32d368e\" (UID: \"84f99285-4558-48e6-b72d-a237e32d368e\") " Jul 08 19:06:34.534771 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534438 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:34.534771 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534650 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:34.534874 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.534689 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.535304 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.535248 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:06:34.537617 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.537592 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:34.538181 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.538144 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:34.538559 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.538507 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-config-out" (OuterVolumeSpecName: "config-out") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:06:34.538664 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.538569 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:34.538664 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.538635 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-config-volume" (OuterVolumeSpecName: "config-volume") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:34.538664 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.538635 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-kube-api-access-9jn67" (OuterVolumeSpecName: "kube-api-access-9jn67") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "kube-api-access-9jn67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:06:34.538830 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.538793 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:34.539044 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.539028 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:06:34.542163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.542135 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:34.548942 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.548921 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-web-config" (OuterVolumeSpecName: "web-config") pod "84f99285-4558-48e6-b72d-a237e32d368e" (UID: "84f99285-4558-48e6-b72d-a237e32d368e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:34.635361 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635326 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-config-volume\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635361 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635354 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-alertmanager-main-db\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635361 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635365 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-cluster-tls-config\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635375 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635386 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84f99285-4558-48e6-b72d-a237e32d368e-metrics-client-ca\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635393 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-tls-assets\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635401 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jn67\" (UniqueName: \"kubernetes.io/projected/84f99285-4558-48e6-b72d-a237e32d368e-kube-api-access-9jn67\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635409 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84f99285-4558-48e6-b72d-a237e32d368e-config-out\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635418 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-web-config\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635425 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635434 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-main-tls\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.635588 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.635442 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84f99285-4558-48e6-b72d-a237e32d368e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:34.887253 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.887154 2569 generic.go:356] "Generic (PLEG): container finished" podID="84f99285-4558-48e6-b72d-a237e32d368e" containerID="e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194" exitCode=0 Jul 08 19:06:34.887405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.887247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194"} Jul 08 19:06:34.887405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.887320 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84f99285-4558-48e6-b72d-a237e32d368e","Type":"ContainerDied","Data":"2c3761a5ff487db3ea387d5263be9ba9db95e9f52cb73753bdda366067d43666"} Jul 08 19:06:34.887405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.887259 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:34.887405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.887355 2569 scope.go:117] "RemoveContainer" containerID="f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495" Jul 08 19:06:34.895071 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.895055 2569 scope.go:117] "RemoveContainer" containerID="e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194" Jul 08 19:06:34.902488 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.902467 2569 scope.go:117] "RemoveContainer" containerID="16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f" Jul 08 19:06:34.908993 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.908973 2569 scope.go:117] "RemoveContainer" containerID="5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad" Jul 08 19:06:34.916017 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.915993 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:06:34.917326 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.917309 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:06:34.917568 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.917534 2569 scope.go:117] "RemoveContainer" containerID="db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0" Jul 08 19:06:34.924898 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.924882 2569 scope.go:117] "RemoveContainer" containerID="8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47" Jul 08 19:06:34.931879 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.931863 2569 scope.go:117] "RemoveContainer" containerID="4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4" Jul 08 19:06:34.938578 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.938562 2569 scope.go:117] "RemoveContainer" containerID="f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495" Jul 08 19:06:34.938854 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:34.938836 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495\": container with ID starting with f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495 not found: ID does not exist" containerID="f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495" Jul 08 19:06:34.938909 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.938864 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495"} err="failed to get container status \"f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495\": rpc error: code = NotFound desc = could not find container \"f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495\": container with ID starting with f2049a1ba9a1156e92a4b7ce0f72e4c6073e40c84b742e80ceef27c9d3034495 not found: ID does not exist" Jul 08 19:06:34.938909 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.938882 2569 scope.go:117] "RemoveContainer" containerID="e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194" Jul 08 19:06:34.939096 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:34.939079 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194\": container with ID starting with e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194 not found: ID does not exist" containerID="e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194" Jul 08 19:06:34.939134 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939101 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194"} err="failed to get container status \"e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194\": rpc error: code = NotFound desc = could not find container \"e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194\": container with ID starting with e9c42184a85918af91f1525601fafbc0ecc0e5121369038c2fb6dbee8d7b5194 not found: ID does not exist" Jul 08 19:06:34.939134 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939116 2569 scope.go:117] "RemoveContainer" containerID="16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f" Jul 08 19:06:34.939351 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:34.939334 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f\": container with ID starting with 16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f not found: ID does not exist" containerID="16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f" Jul 08 19:06:34.939422 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939360 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f"} err="failed to get container status \"16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f\": rpc error: code = NotFound desc = could not find container \"16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f\": container with ID starting with 16b588ee602d9527ad83a7db24cdbfabbb85b31f659b72924d17f4bb0b0dae9f not found: ID does not exist" Jul 08 19:06:34.939422 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939380 2569 scope.go:117] "RemoveContainer" containerID="5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad" Jul 08 19:06:34.939652 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:34.939635 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad\": container with ID starting with 5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad not found: ID does not exist" containerID="5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad" Jul 08 19:06:34.939724 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939659 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad"} err="failed to get container status \"5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad\": rpc error: code = NotFound desc = could not find container \"5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad\": container with ID starting with 5be4de68dc279025c540f7338fc78c7332f8df95b100619b50e90b70c6f6bdad not found: ID does not exist" Jul 08 19:06:34.939724 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939681 2569 scope.go:117] "RemoveContainer" containerID="db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0" Jul 08 19:06:34.939921 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:34.939905 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0\": container with ID starting with db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0 not found: ID does not exist" containerID="db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0" Jul 08 19:06:34.939963 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939929 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0"} err="failed to get container status \"db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0\": rpc error: code = NotFound desc = could not find container \"db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0\": container with ID starting with db283aa902861e82910852a3a1b8a36f79d94caeccb6c861679c8f059169f2a0 not found: ID does not exist" Jul 08 19:06:34.939963 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.939945 2569 scope.go:117] "RemoveContainer" containerID="8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47" Jul 08 19:06:34.940153 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:34.940139 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47\": container with ID starting with 8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47 not found: ID does not exist" containerID="8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47" Jul 08 19:06:34.940200 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.940154 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47"} err="failed to get container status \"8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47\": rpc error: code = NotFound desc = could not find container \"8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47\": container with ID starting with 8c989d2d5103f02656902f8c73f5f81f0902c907c3360821e5699bc601931c47 not found: ID does not exist" Jul 08 19:06:34.940200 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.940164 2569 scope.go:117] "RemoveContainer" containerID="4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4" Jul 08 19:06:34.940357 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:34.940341 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4\": container with ID starting with 4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4 not found: ID does not exist" containerID="4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4" Jul 08 19:06:34.940406 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.940365 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4"} err="failed to get container status \"4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4\": rpc error: code = NotFound desc = could not find container \"4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4\": container with ID starting with 4c308d47b5a812ed2044ac3c913cb8ce3d1d8b443e6c48a39314d00bb4ee8bb4 not found: ID does not exist" Jul 08 19:06:34.961797 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.961775 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:06:34.962263 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962242 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="alertmanager" Jul 08 19:06:34.962347 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962267 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="alertmanager" Jul 08 19:06:34.962347 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962281 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="prom-label-proxy" Jul 08 19:06:34.962347 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962289 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="prom-label-proxy" Jul 08 19:06:34.962347 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962305 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-web" Jul 08 19:06:34.962347 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962314 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-web" Jul 08 19:06:34.962347 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962323 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy" Jul 08 19:06:34.962347 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962332 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962356 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="init-config-reloader" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962365 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="init-config-reloader" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962376 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-metric" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962385 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-metric" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962398 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97092863-40a5-4bc4-822c-22c47d6f1d96" containerName="console" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962408 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97092863-40a5-4bc4-822c-22c47d6f1d96" containerName="console" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962419 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="config-reloader" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962427 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="config-reloader" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962553 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="config-reloader" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962567 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-metric" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962577 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="prom-label-proxy" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962586 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="alertmanager" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962598 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy-web" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962606 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f99285-4558-48e6-b72d-a237e32d368e" containerName="kube-rbac-proxy" Jul 08 19:06:34.962716 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.962614 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97092863-40a5-4bc4-822c-22c47d6f1d96" containerName="console" Jul 08 19:06:34.966715 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.966688 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:34.971525 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.971507 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 19:06:34.972635 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.972615 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 19:06:34.973325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.973149 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 19:06:34.973325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.973186 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tc6fb\"" Jul 08 19:06:34.973325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.973194 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 19:06:34.973325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.973204 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 19:06:34.973325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.973196 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 19:06:34.973666 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.973513 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 19:06:34.973666 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.973523 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 19:06:34.980111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.980093 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 19:06:34.989925 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:34.989897 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:06:35.039640 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039608 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp68p\" (UniqueName: \"kubernetes.io/projected/ca1fb528-0921-4044-90a7-134c42f800b9-kube-api-access-kp68p\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039640 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca1fb528-0921-4044-90a7-134c42f800b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039816 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039816 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039710 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039816 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039816 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039782 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca1fb528-0921-4044-90a7-134c42f800b9-config-out\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039816 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca1fb528-0921-4044-90a7-134c42f800b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca1fb528-0921-4044-90a7-134c42f800b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039841 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca1fb528-0921-4044-90a7-134c42f800b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039886 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039909 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-web-config\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.039962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.039956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.097775 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.097742 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f99285-4558-48e6-b72d-a237e32d368e" path="/var/lib/kubelet/pods/84f99285-4558-48e6-b72d-a237e32d368e/volumes" Jul 08 19:06:35.141215 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141215 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp68p\" (UniqueName: \"kubernetes.io/projected/ca1fb528-0921-4044-90a7-134c42f800b9-kube-api-access-kp68p\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141215 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141211 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca1fb528-0921-4044-90a7-134c42f800b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141436 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141232 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141436 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141436 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141436 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca1fb528-0921-4044-90a7-134c42f800b9-config-out\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141436 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca1fb528-0921-4044-90a7-134c42f800b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141711 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca1fb528-0921-4044-90a7-134c42f800b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141711 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141711 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca1fb528-0921-4044-90a7-134c42f800b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141711 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141582 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141711 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-web-config\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.141949 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.141844 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca1fb528-0921-4044-90a7-134c42f800b9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.142441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.142412 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca1fb528-0921-4044-90a7-134c42f800b9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.142441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.142421 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca1fb528-0921-4044-90a7-134c42f800b9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.144656 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.144606 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca1fb528-0921-4044-90a7-134c42f800b9-config-out\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.144656 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.144612 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.144799 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.144688 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.144877 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.144853 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.145002 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.144853 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.145085 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.145068 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.145311 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.145200 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-web-config\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.145615 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.145597 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca1fb528-0921-4044-90a7-134c42f800b9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.146628 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.146612 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca1fb528-0921-4044-90a7-134c42f800b9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.150992 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.150977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp68p\" (UniqueName: \"kubernetes.io/projected/ca1fb528-0921-4044-90a7-134c42f800b9-kube-api-access-kp68p\") pod \"alertmanager-main-0\" (UID: \"ca1fb528-0921-4044-90a7-134c42f800b9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.275311 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.275283 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 19:06:35.450773 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.450746 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 19:06:35.452269 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:06:35.452246 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1fb528_0921_4044_90a7_134c42f800b9.slice/crio-db8075d2af681a4345df2852ebe2b977efaf29c3288f302f1fe01c75055910e0 WatchSource:0}: Error finding container db8075d2af681a4345df2852ebe2b977efaf29c3288f302f1fe01c75055910e0: Status 404 returned error can't find the container with id db8075d2af681a4345df2852ebe2b977efaf29c3288f302f1fe01c75055910e0 Jul 08 19:06:35.892969 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.892878 2569 generic.go:356] "Generic (PLEG): container finished" podID="ca1fb528-0921-4044-90a7-134c42f800b9" containerID="0fcab79a36b9400c63b1d12f0799db3db4b70cb82d5845c1d98c02acec8df8b4" exitCode=0 Jul 08 19:06:35.893125 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.892972 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerDied","Data":"0fcab79a36b9400c63b1d12f0799db3db4b70cb82d5845c1d98c02acec8df8b4"} Jul 08 19:06:35.893125 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:35.893049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerStarted","Data":"db8075d2af681a4345df2852ebe2b977efaf29c3288f302f1fe01c75055910e0"} Jul 08 19:06:36.901292 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:36.901262 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerStarted","Data":"6b9c053b44d03c90accff8ac07fd712e20ac1e0ef6d3c6a847575e3bc7bbc9c3"} Jul 08 19:06:36.901637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:36.901330 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerStarted","Data":"e3eb6979a13c5a32c2fd31b90c4d7dceda9f4b1d7365fad318e34c2e1fa0cf7a"} Jul 08 19:06:36.901637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:36.901356 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerStarted","Data":"bdf2beb0d66ae9dbca07a091fb99e931316edf7e88d38ea0b9dfefe84f93a6be"} Jul 08 19:06:36.901637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:36.901388 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerStarted","Data":"67d57433152bf5ea528762a712b38ac146053cf54faccc6393b849865c3b822b"} Jul 08 19:06:36.901637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:36.901415 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerStarted","Data":"0e4060310de09859eb22ab935fa3a4971bb09b222424d6592c2e95359f8816be"} Jul 08 19:06:36.901637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:36.901443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca1fb528-0921-4044-90a7-134c42f800b9","Type":"ContainerStarted","Data":"e561b9cbec9e8e6702d73feeec936d07250f22c8310590ac32cbf3524898611e"} Jul 08 19:06:36.938730 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:36.938679 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.921315277 podStartE2EDuration="2.938662874s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.347597ms" isStatefulPod=false podCreationTimestamp="2026-07-08 19:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:06:36.937225839 +0000 UTC m=+142.376528486" watchObservedRunningTime="2026-07-08 19:06:36.938662874 +0000 UTC m=+142.377965505" Jul 08 19:06:37.314141 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.314106 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt"] Jul 08 19:06:37.316989 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.316966 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.320966 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.320934 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zj5jn\"" Jul 08 19:06:37.320966 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.320959 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 08 19:06:37.321163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.320933 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 08 19:06:37.321410 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.321384 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 08 19:06:37.321513 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.321425 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 08 19:06:37.321848 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.321819 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 08 19:06:37.326927 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.326908 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 08 19:06:37.341176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.341140 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt"] Jul 08 19:06:37.360556 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360510 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-secret-telemeter-client\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.360686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-federate-client-tls\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.360686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-metrics-client-ca\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.360686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.360686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57rbk\" (UniqueName: \"kubernetes.io/projected/c3c36089-0bef-4f8f-a926-3f5a165e376c-kube-api-access-57rbk\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.360686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360676 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-telemeter-client-tls\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.360855 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-serving-certs-ca-bundle\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.360855 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.360726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.447350 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.447315 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:06:37.447641 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.447612 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="prometheus" containerID="cri-o://aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" gracePeriod=600 Jul 08 19:06:37.447736 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.447635 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy" containerID="cri-o://c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" gracePeriod=600 Jul 08 19:06:37.447736 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.447681 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="config-reloader" containerID="cri-o://3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" gracePeriod=600 Jul 08 19:06:37.447736 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.447695 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-web" containerID="cri-o://f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" gracePeriod=600 Jul 08 19:06:37.447882 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.447698 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="thanos-sidecar" containerID="cri-o://e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" gracePeriod=600 Jul 08 19:06:37.447882 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.447767 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" gracePeriod=600 Jul 08 19:06:37.461369 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-secret-telemeter-client\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.461502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-federate-client-tls\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.461502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-metrics-client-ca\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.461502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.461502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57rbk\" (UniqueName: \"kubernetes.io/projected/c3c36089-0bef-4f8f-a926-3f5a165e376c-kube-api-access-57rbk\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.461802 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461516 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-telemeter-client-tls\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.461802 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-serving-certs-ca-bundle\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.461802 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.461622 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.462905 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.462872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.463297 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.463268 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-serving-certs-ca-bundle\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.463798 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.463772 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3c36089-0bef-4f8f-a926-3f5a165e376c-metrics-client-ca\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.464977 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.464952 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-secret-telemeter-client\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.468533 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.465395 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.468533 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.466271 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-telemeter-client-tls\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.468533 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.467188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3c36089-0bef-4f8f-a926-3f5a165e376c-federate-client-tls\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.471762 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.471735 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57rbk\" (UniqueName: \"kubernetes.io/projected/c3c36089-0bef-4f8f-a926-3f5a165e376c-kube-api-access-57rbk\") pod \"telemeter-client-b74f6f7dc-dfsvt\" (UID: \"c3c36089-0bef-4f8f-a926-3f5a165e376c\") " pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.628942 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.628289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" Jul 08 19:06:37.718765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.718730 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:37.766221 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.765808 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-tls-assets\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.766221 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.765851 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.766221 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.765879 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-metrics-client-certs\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.766221 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.765903 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-2\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.766221 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.765944 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-grpc-tls\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.766221 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.765980 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.766221 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.766009 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-db\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.766696 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.766522 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:37.767877 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.767832 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:06:37.768013 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.767978 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-web-config\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768071 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768017 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv5f4\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-kube-api-access-dv5f4\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768071 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768060 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-trusted-ca-bundle\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768094 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-kube-rbac-proxy\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768123 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-config\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768172 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768151 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-serving-certs-ca-bundle\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768300 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768202 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-0\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768300 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768227 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-thanos-prometheus-http-client-file\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768300 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768261 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-config-out\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768300 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768288 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-kubelet-serving-ca-bundle\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768473 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768315 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-tls\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768473 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768362 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-1\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.768473 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768392 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-metrics-client-ca\") pod \"14bef7f6-39ad-48d7-9317-cb630e70730b\" (UID: \"14bef7f6-39ad-48d7-9317-cb630e70730b\") " Jul 08 19:06:37.769128 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768712 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-db\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.769128 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.768736 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.770154 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.769353 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:37.770154 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.769755 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:37.770154 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.770139 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:37.770925 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.770526 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:37.771048 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.770926 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.771048 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.770972 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:37.772366 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.772306 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.772558 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.772512 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:06:37.773204 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.772913 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.777333 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.777302 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:06:37.777512 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.777458 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.777512 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.777479 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-config-out" (OuterVolumeSpecName: "config-out") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:06:37.778116 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.777972 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-kube-api-access-dv5f4" (OuterVolumeSpecName: "kube-api-access-dv5f4") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "kube-api-access-dv5f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:06:37.778116 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.777984 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.778260 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.778139 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.778493 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.778451 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.778983 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.778961 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-config" (OuterVolumeSpecName: "config") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.792441 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.792400 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-web-config" (OuterVolumeSpecName: "web-config") pod "14bef7f6-39ad-48d7-9317-cb630e70730b" (UID: "14bef7f6-39ad-48d7-9317-cb630e70730b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:06:37.812408 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.812376 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt"] Jul 08 19:06:37.813213 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:06:37.813177 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c36089_0bef_4f8f_a926_3f5a165e376c.slice/crio-83f3eebbf618ba8967f4172a586299b545098ae942090ad2506d235330ecc066 WatchSource:0}: Error finding container 83f3eebbf618ba8967f4172a586299b545098ae942090ad2506d235330ecc066: Status 404 returned error can't find the container with id 83f3eebbf618ba8967f4172a586299b545098ae942090ad2506d235330ecc066 Jul 08 19:06:37.870016 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.869972 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870016 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870015 2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870034 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14bef7f6-39ad-48d7-9317-cb630e70730b-config-out\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870049 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870062 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870079 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870094 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-metrics-client-ca\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870107 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-tls-assets\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870120 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870134 2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-metrics-client-certs\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870149 2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-grpc-tls\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870163 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870176 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-web-config\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870189 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv5f4\" (UniqueName: \"kubernetes.io/projected/14bef7f6-39ad-48d7-9317-cb630e70730b-kube-api-access-dv5f4\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870204 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870217 2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-secret-kube-rbac-proxy\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870231 2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/14bef7f6-39ad-48d7-9317-cb630e70730b-config\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.870697 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.870245 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14bef7f6-39ad-48d7-9317-cb630e70730b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:06:37.910499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910462 2569 generic.go:356] "Generic (PLEG): container finished" podID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" exitCode=0 Jul 08 19:06:37.910499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910493 2569 generic.go:356] "Generic (PLEG): container finished" podID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" exitCode=0 Jul 08 19:06:37.910499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910504 2569 generic.go:356] "Generic (PLEG): container finished" podID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" exitCode=0 Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910511 2569 generic.go:356] "Generic (PLEG): container finished" podID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" exitCode=0 Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910520 2569 generic.go:356] "Generic (PLEG): container finished" podID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" exitCode=0 Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910527 2569 generic.go:356] "Generic (PLEG): container finished" podID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" exitCode=0 Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910667 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910737 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910775 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910891 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910930 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14bef7f6-39ad-48d7-9317-cb630e70730b","Type":"ContainerDied","Data":"770fbc8ff294ff5dcc98b1fe87ef8865d5aa052d19bd09fd40789aa907569187"} Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.910957 2569 scope.go:117] "RemoveContainer" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.911069 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.911041 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:37.912872 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.912842 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" event={"ID":"c3c36089-0bef-4f8f-a926-3f5a165e376c","Type":"ContainerStarted","Data":"83f3eebbf618ba8967f4172a586299b545098ae942090ad2506d235330ecc066"} Jul 08 19:06:37.920376 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.920355 2569 scope.go:117] "RemoveContainer" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.928968 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.928946 2569 scope.go:117] "RemoveContainer" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.937292 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.937269 2569 scope.go:117] "RemoveContainer" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.941115 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.941087 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:06:37.946190 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.946162 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:06:37.948031 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.948013 2569 scope.go:117] "RemoveContainer" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.956554 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.956522 2569 scope.go:117] "RemoveContainer" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.966438 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.965342 2569 scope.go:117] "RemoveContainer" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.974732 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.974703 2569 scope.go:117] "RemoveContainer" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.975030 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:37.975011 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": container with ID starting with 040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15 not found: ID does not exist" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.975101 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.975039 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} err="failed to get container status \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": rpc error: code = NotFound desc = could not find container \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": container with ID starting with 040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15 not found: ID does not exist" Jul 08 19:06:37.975101 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.975057 2569 scope.go:117] "RemoveContainer" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.975338 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:37.975320 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": container with ID starting with c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27 not found: ID does not exist" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.975384 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.975341 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} err="failed to get container status \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": rpc error: code = NotFound desc = could not find container \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": container with ID starting with c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27 not found: ID does not exist" Jul 08 19:06:37.975384 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.975356 2569 scope.go:117] "RemoveContainer" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.975721 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:37.975701 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": container with ID starting with f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41 not found: ID does not exist" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.975810 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.975730 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} err="failed to get container status \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": rpc error: code = NotFound desc = could not find container \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": container with ID starting with f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41 not found: ID does not exist" Jul 08 19:06:37.975810 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.975752 2569 scope.go:117] "RemoveContainer" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.975996 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:37.975977 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": container with ID starting with e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7 not found: ID does not exist" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.976035 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976000 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} err="failed to get container status \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": rpc error: code = NotFound desc = could not find container \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": container with ID starting with e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7 not found: ID does not exist" Jul 08 19:06:37.976035 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976014 2569 scope.go:117] "RemoveContainer" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.976242 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:37.976222 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": container with ID starting with 3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f not found: ID does not exist" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.976329 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976246 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} err="failed to get container status \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": rpc error: code = NotFound desc = could not find container \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": container with ID starting with 3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f not found: ID does not exist" Jul 08 19:06:37.976329 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976262 2569 scope.go:117] "RemoveContainer" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.976497 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:37.976475 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": container with ID starting with aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d not found: ID does not exist" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.976558 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976508 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} err="failed to get container status \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": rpc error: code = NotFound desc = could not find container \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": container with ID starting with aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d not found: ID does not exist" Jul 08 19:06:37.976558 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976529 2569 scope.go:117] "RemoveContainer" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.976814 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:06:37.976796 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": container with ID starting with 6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd not found: ID does not exist" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.976861 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976819 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd"} err="failed to get container status \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": rpc error: code = NotFound desc = could not find container \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": container with ID starting with 6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd not found: ID does not exist" Jul 08 19:06:37.976861 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.976834 2569 scope.go:117] "RemoveContainer" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.977076 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977055 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} err="failed to get container status \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": rpc error: code = NotFound desc = could not find container \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": container with ID starting with 040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15 not found: ID does not exist" Jul 08 19:06:37.977076 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977077 2569 scope.go:117] "RemoveContainer" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.977365 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977339 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} err="failed to get container status \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": rpc error: code = NotFound desc = could not find container \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": container with ID starting with c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27 not found: ID does not exist" Jul 08 19:06:37.977464 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977366 2569 scope.go:117] "RemoveContainer" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.977656 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977630 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} err="failed to get container status \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": rpc error: code = NotFound desc = could not find container \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": container with ID starting with f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41 not found: ID does not exist" Jul 08 19:06:37.977764 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977657 2569 scope.go:117] "RemoveContainer" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.977906 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977887 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} err="failed to get container status \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": rpc error: code = NotFound desc = could not find container \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": container with ID starting with e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7 not found: ID does not exist" Jul 08 19:06:37.977962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.977908 2569 scope.go:117] "RemoveContainer" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.978159 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.978136 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} err="failed to get container status \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": rpc error: code = NotFound desc = could not find container \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": container with ID starting with 3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f not found: ID does not exist" Jul 08 19:06:37.978159 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.978157 2569 scope.go:117] "RemoveContainer" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.978480 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.978425 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} err="failed to get container status \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": rpc error: code = NotFound desc = could not find container \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": container with ID starting with aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d not found: ID does not exist" Jul 08 19:06:37.978480 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.978448 2569 scope.go:117] "RemoveContainer" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.978817 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.978717 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd"} err="failed to get container status \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": rpc error: code = NotFound desc = could not find container \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": container with ID starting with 6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd not found: ID does not exist" Jul 08 19:06:37.978817 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.978753 2569 scope.go:117] "RemoveContainer" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.979078 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.978998 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} err="failed to get container status \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": rpc error: code = NotFound desc = could not find container \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": container with ID starting with 040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15 not found: ID does not exist" Jul 08 19:06:37.979078 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.979020 2569 scope.go:117] "RemoveContainer" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.979328 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.979245 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} err="failed to get container status \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": rpc error: code = NotFound desc = could not find container \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": container with ID starting with c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27 not found: ID does not exist" Jul 08 19:06:37.979328 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.979268 2569 scope.go:117] "RemoveContainer" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.979587 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.979498 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} err="failed to get container status \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": rpc error: code = NotFound desc = could not find container \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": container with ID starting with f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41 not found: ID does not exist" Jul 08 19:06:37.979587 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.979517 2569 scope.go:117] "RemoveContainer" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.979817 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.979795 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} err="failed to get container status \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": rpc error: code = NotFound desc = could not find container \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": container with ID starting with e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7 not found: ID does not exist" Jul 08 19:06:37.979817 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.979817 2569 scope.go:117] "RemoveContainer" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.980056 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980038 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} err="failed to get container status \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": rpc error: code = NotFound desc = could not find container \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": container with ID starting with 3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f not found: ID does not exist" Jul 08 19:06:37.980107 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980058 2569 scope.go:117] "RemoveContainer" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.980249 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980232 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} err="failed to get container status \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": rpc error: code = NotFound desc = could not find container \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": container with ID starting with aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d not found: ID does not exist" Jul 08 19:06:37.980331 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980250 2569 scope.go:117] "RemoveContainer" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.980442 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980427 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd"} err="failed to get container status \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": rpc error: code = NotFound desc = could not find container \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": container with ID starting with 6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd not found: ID does not exist" Jul 08 19:06:37.980479 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980442 2569 scope.go:117] "RemoveContainer" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.980725 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980695 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} err="failed to get container status \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": rpc error: code = NotFound desc = could not find container \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": container with ID starting with 040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15 not found: ID does not exist" Jul 08 19:06:37.980799 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.980726 2569 scope.go:117] "RemoveContainer" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.981071 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981049 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} err="failed to get container status \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": rpc error: code = NotFound desc = could not find container \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": container with ID starting with c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27 not found: ID does not exist" Jul 08 19:06:37.981142 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981072 2569 scope.go:117] "RemoveContainer" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.981313 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981296 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} err="failed to get container status \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": rpc error: code = NotFound desc = could not find container \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": container with ID starting with f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41 not found: ID does not exist" Jul 08 19:06:37.981369 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981314 2569 scope.go:117] "RemoveContainer" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.981525 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981511 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} err="failed to get container status \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": rpc error: code = NotFound desc = could not find container \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": container with ID starting with e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7 not found: ID does not exist" Jul 08 19:06:37.981525 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981526 2569 scope.go:117] "RemoveContainer" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.981841 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981821 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} err="failed to get container status \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": rpc error: code = NotFound desc = could not find container \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": container with ID starting with 3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f not found: ID does not exist" Jul 08 19:06:37.981906 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.981843 2569 scope.go:117] "RemoveContainer" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.982189 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.982170 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} err="failed to get container status \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": rpc error: code = NotFound desc = could not find container \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": container with ID starting with aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d not found: ID does not exist" Jul 08 19:06:37.982295 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.982191 2569 scope.go:117] "RemoveContainer" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.982618 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.982598 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd"} err="failed to get container status \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": rpc error: code = NotFound desc = could not find container \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": container with ID starting with 6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd not found: ID does not exist" Jul 08 19:06:37.982701 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.982618 2569 scope.go:117] "RemoveContainer" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.982891 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.982872 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} err="failed to get container status \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": rpc error: code = NotFound desc = could not find container \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": container with ID starting with 040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15 not found: ID does not exist" Jul 08 19:06:37.982933 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.982894 2569 scope.go:117] "RemoveContainer" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.983159 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983140 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} err="failed to get container status \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": rpc error: code = NotFound desc = could not find container \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": container with ID starting with c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27 not found: ID does not exist" Jul 08 19:06:37.983212 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983160 2569 scope.go:117] "RemoveContainer" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.983402 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983385 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} err="failed to get container status \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": rpc error: code = NotFound desc = could not find container \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": container with ID starting with f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41 not found: ID does not exist" Jul 08 19:06:37.983445 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983404 2569 scope.go:117] "RemoveContainer" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.983643 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983624 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} err="failed to get container status \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": rpc error: code = NotFound desc = could not find container \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": container with ID starting with e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7 not found: ID does not exist" Jul 08 19:06:37.983721 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983655 2569 scope.go:117] "RemoveContainer" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.983721 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983693 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:06:37.983935 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983917 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} err="failed to get container status \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": rpc error: code = NotFound desc = could not find container \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": container with ID starting with 3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f not found: ID does not exist" Jul 08 19:06:37.983984 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.983935 2569 scope.go:117] "RemoveContainer" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.984084 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984066 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="thanos-sidecar" Jul 08 19:06:37.984129 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984088 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="thanos-sidecar" Jul 08 19:06:37.984129 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984099 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-web" Jul 08 19:06:37.984129 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984108 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-web" Jul 08 19:06:37.984129 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984120 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy" Jul 08 19:06:37.984129 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984128 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984143 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="init-config-reloader" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984149 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="init-config-reloader" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984155 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="config-reloader" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984164 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="config-reloader" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984176 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-thanos" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984181 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-thanos" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984189 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="prometheus" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984195 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="prometheus" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984229 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} err="failed to get container status \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": rpc error: code = NotFound desc = could not find container \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": container with ID starting with aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d not found: ID does not exist" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984249 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="prometheus" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984252 2569 scope.go:117] "RemoveContainer" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984256 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="config-reloader" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984323 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984335 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="thanos-sidecar" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984346 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-thanos" Jul 08 19:06:37.984344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984353 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" containerName="kube-rbac-proxy-web" Jul 08 19:06:37.985119 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984517 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd"} err="failed to get container status \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": rpc error: code = NotFound desc = could not find container \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": container with ID starting with 6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd not found: ID does not exist" Jul 08 19:06:37.985119 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984622 2569 scope.go:117] "RemoveContainer" containerID="040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15" Jul 08 19:06:37.985119 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984954 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15"} err="failed to get container status \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": rpc error: code = NotFound desc = could not find container \"040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15\": container with ID starting with 040fb03447e0698627c5978ca37741a7be9858d6e5fb52cccc4e244651eabe15 not found: ID does not exist" Jul 08 19:06:37.985119 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.984982 2569 scope.go:117] "RemoveContainer" containerID="c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27" Jul 08 19:06:37.985308 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.985252 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27"} err="failed to get container status \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": rpc error: code = NotFound desc = could not find container \"c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27\": container with ID starting with c83d79211a5187462f35d6b8df6503aa567f2c743c714fee8b30be5dc67a4f27 not found: ID does not exist" Jul 08 19:06:37.985308 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.985284 2569 scope.go:117] "RemoveContainer" containerID="f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41" Jul 08 19:06:37.985575 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.985510 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41"} err="failed to get container status \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": rpc error: code = NotFound desc = could not find container \"f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41\": container with ID starting with f9c7363fb2e2b33bf6441b834265d267dfcc279453116bf70fede9856581ef41 not found: ID does not exist" Jul 08 19:06:37.985641 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.985577 2569 scope.go:117] "RemoveContainer" containerID="e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7" Jul 08 19:06:37.985796 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.985775 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7"} err="failed to get container status \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": rpc error: code = NotFound desc = could not find container \"e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7\": container with ID starting with e0fce19bbf52369c94daae034f3a0e38bda9970c9514940e47dee3af43f4d8b7 not found: ID does not exist" Jul 08 19:06:37.985854 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.985796 2569 scope.go:117] "RemoveContainer" containerID="3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f" Jul 08 19:06:37.986000 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.985979 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f"} err="failed to get container status \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": rpc error: code = NotFound desc = could not find container \"3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f\": container with ID starting with 3b8d52efc6baf196a0764f8a149e81936bfa3b300bcaeb7bf94555f59efe304f not found: ID does not exist" Jul 08 19:06:37.986054 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.986003 2569 scope.go:117] "RemoveContainer" containerID="aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d" Jul 08 19:06:37.986206 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.986189 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d"} err="failed to get container status \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": rpc error: code = NotFound desc = could not find container \"aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d\": container with ID starting with aec6a40cfb8775d0f3b330efa182dff36cc9d5deb162c17137fea11098ef340d not found: ID does not exist" Jul 08 19:06:37.986267 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.986206 2569 scope.go:117] "RemoveContainer" containerID="6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd" Jul 08 19:06:37.986409 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.986386 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd"} err="failed to get container status \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": rpc error: code = NotFound desc = could not find container \"6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd\": container with ID starting with 6ebd22976f4873408409fb275c1f4d403f06e36b792e829a3da34b904da125cd not found: ID does not exist" Jul 08 19:06:37.989175 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.989158 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:37.992916 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.992617 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 19:06:37.996500 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.996472 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 19:06:37.996718 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.996609 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 19:06:37.996793 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.996743 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 19:06:37.996846 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.996824 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 19:06:37.996904 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.996843 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3rn0nhuf4d360\"" Jul 08 19:06:37.996904 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.996875 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 19:06:37.997064 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.996979 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 19:06:37.997207 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.997081 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 19:06:37.997636 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.997614 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-px2wp\"" Jul 08 19:06:37.997754 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.997736 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 19:06:37.997906 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.997885 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 19:06:37.998141 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.998125 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 19:06:37.998226 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.998210 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 19:06:37.999596 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:37.999573 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 19:06:38.004954 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.004935 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 19:06:38.011854 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.011830 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:06:38.072513 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-web-config\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072513 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072513 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072513 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072612 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c24f2587-9388-47a9-a5ac-e52d66c77b82-config-out\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072666 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072684 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.072781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072832 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-config\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072856 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c24f2587-9388-47a9-a5ac-e52d66c77b82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072877 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.072967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.073001 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmmc\" (UniqueName: \"kubernetes.io/projected/c24f2587-9388-47a9-a5ac-e52d66c77b82-kube-api-access-8vmmc\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.073197 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.073021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.173612 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.173612 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.173813 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.173813 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmmc\" (UniqueName: \"kubernetes.io/projected/c24f2587-9388-47a9-a5ac-e52d66c77b82-kube-api-access-8vmmc\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.173889 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173823 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.173889 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-web-config\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174015 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173912 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174015 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174015 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173968 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174015 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.173991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174203 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c24f2587-9388-47a9-a5ac-e52d66c77b82-config-out\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174203 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174071 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174203 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174117 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174203 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174203 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174234 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-config\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c24f2587-9388-47a9-a5ac-e52d66c77b82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174403 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.174930 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174844 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.175630 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.175604 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.176318 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.176286 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.176674 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.176512 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-web-config\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.176674 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.174231 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.176975 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.176958 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.177499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.177176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.177499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.177239 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.177499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.177312 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.177705 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.177634 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.177705 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.177657 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.178389 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.178335 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c24f2587-9388-47a9-a5ac-e52d66c77b82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.178946 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.178917 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c24f2587-9388-47a9-a5ac-e52d66c77b82-config-out\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.179212 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.179192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.179585 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.179532 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-config\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.179661 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.179646 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.180005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.179988 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c24f2587-9388-47a9-a5ac-e52d66c77b82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.180062 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.180004 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c24f2587-9388-47a9-a5ac-e52d66c77b82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.182623 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.182607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmmc\" (UniqueName: \"kubernetes.io/projected/c24f2587-9388-47a9-a5ac-e52d66c77b82-kube-api-access-8vmmc\") pod \"prometheus-k8s-0\" (UID: \"c24f2587-9388-47a9-a5ac-e52d66c77b82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.300462 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.300411 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:06:38.453631 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.453594 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 19:06:38.455724 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:06:38.455698 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24f2587_9388_47a9_a5ac_e52d66c77b82.slice/crio-61f157f2fb0f88a15e4414c590a943945c0b124341e51398ce8eda96e7be0515 WatchSource:0}: Error finding container 61f157f2fb0f88a15e4414c590a943945c0b124341e51398ce8eda96e7be0515: Status 404 returned error can't find the container with id 61f157f2fb0f88a15e4414c590a943945c0b124341e51398ce8eda96e7be0515 Jul 08 19:06:38.919830 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.919800 2569 generic.go:356] "Generic (PLEG): container finished" podID="c24f2587-9388-47a9-a5ac-e52d66c77b82" containerID="e1939a86782e77e9a94a8d23243279a8ea1184acff8671f6ccc1455c1782589f" exitCode=0 Jul 08 19:06:38.920211 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.919891 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerDied","Data":"e1939a86782e77e9a94a8d23243279a8ea1184acff8671f6ccc1455c1782589f"} Jul 08 19:06:38.920211 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:38.919959 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerStarted","Data":"61f157f2fb0f88a15e4414c590a943945c0b124341e51398ce8eda96e7be0515"} Jul 08 19:06:39.101229 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.101193 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14bef7f6-39ad-48d7-9317-cb630e70730b" path="/var/lib/kubelet/pods/14bef7f6-39ad-48d7-9317-cb630e70730b/volumes" Jul 08 19:06:39.927934 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.927896 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerStarted","Data":"1d25d9ac9eeac135b7112d0bb77042943a2ef007888aaecd6efe9dd82da358ff"} Jul 08 19:06:39.928355 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.927962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerStarted","Data":"510353bc32bedb86caf5b50265834c528d96e34f5c6cdd67794edee5d9280320"} Jul 08 19:06:39.928355 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.927993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerStarted","Data":"f484d2024f96165c4657f0eb7b76e5015f447de8ef3d1d9c0b3e11383c436f26"} Jul 08 19:06:39.928355 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.928019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerStarted","Data":"a6a63cafe6f35c8d8850b1419230e5d7f55ebb0c76f088edc4a2154d0faa932e"} Jul 08 19:06:39.928355 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.928042 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerStarted","Data":"1cfa66f4b885403044e3d241f68f82205b1e89fb94932b3245362f1a37a2354a"} Jul 08 19:06:39.928355 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.928082 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c24f2587-9388-47a9-a5ac-e52d66c77b82","Type":"ContainerStarted","Data":"3aedd62b10e943384f68d89e65ce1c4767c7a2955796f33711aac362aaf6bb03"} Jul 08 19:06:39.984419 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:39.984356 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.961079344 podStartE2EDuration="2.984334775s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.255431ms" isStatefulPod=false podCreationTimestamp="2026-07-08 19:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:06:39.982044044 +0000 UTC m=+145.421346702" watchObservedRunningTime="2026-07-08 19:06:39.984334775 +0000 UTC m=+145.423637406" Jul 08 19:06:40.933800 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:40.933760 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" event={"ID":"c3c36089-0bef-4f8f-a926-3f5a165e376c","Type":"ContainerStarted","Data":"70849d7fb6a1bde3bae753e7cef38b85cc624ac9e495a5ebb6d5adb18652189b"} Jul 08 19:06:40.934210 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:40.933850 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" event={"ID":"c3c36089-0bef-4f8f-a926-3f5a165e376c","Type":"ContainerStarted","Data":"fe2fd961d83ba6bff8fd02aac1ac8ae4cdddce7d5dbaa97612cc9cacdf85445c"} Jul 08 19:06:40.934210 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:40.933879 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" event={"ID":"c3c36089-0bef-4f8f-a926-3f5a165e376c","Type":"ContainerStarted","Data":"a8e6ac435bd4008a20b72b0f247fb6ba4351c9b8f42640715694fcc13e15052e"} Jul 08 19:06:43.300590 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:06:43.300513 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:07:38.301627 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:07:38.301587 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:07:38.317757 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:07:38.317732 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:07:38.349216 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:07:38.349142 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b74f6f7dc-dfsvt" podStartSLOduration=58.665819454 podStartE2EDuration="1m1.349119648s" totalImagesPullingTime="2.683300194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:06:40.966816709 +0000 UTC m=+146.406119340" watchObservedRunningTime="2026-07-08 19:07:38.349119648 +0000 UTC m=+203.788422280" Jul 08 19:07:39.179913 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:07:39.179885 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 19:09:14.927800 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:14.927767 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:09:14.935124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:14.935100 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:09:14.935826 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:14.935803 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 19:09:18.128027 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.127992 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg"] Jul 08 19:09:18.130102 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.130087 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.132430 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.132406 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 19:09:18.132670 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.132651 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:09:18.132785 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.132667 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gh9fh\"" Jul 08 19:09:18.140124 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.140106 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg"] Jul 08 19:09:18.222451 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.222419 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vd6hn"] Jul 08 19:09:18.224627 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.224608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.227212 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.227193 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 19:09:18.227294 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.227216 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gr9bz\"" Jul 08 19:09:18.227294 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.227193 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 19:09:18.236140 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.236120 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vd6hn"] Jul 08 19:09:18.290001 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.289966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8808ab32-7df7-45ea-9abb-924ab62e8728-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hn6hg\" (UID: \"8808ab32-7df7-45ea-9abb-924ab62e8728\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.290158 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.290038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cx8c\" (UniqueName: \"kubernetes.io/projected/8808ab32-7df7-45ea-9abb-924ab62e8728-kube-api-access-2cx8c\") pod \"openshift-lws-operator-fd8ccff4c-hn6hg\" (UID: \"8808ab32-7df7-45ea-9abb-924ab62e8728\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.390919 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.390844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8808ab32-7df7-45ea-9abb-924ab62e8728-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hn6hg\" (UID: \"8808ab32-7df7-45ea-9abb-924ab62e8728\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.390919 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.390898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cx8c\" (UniqueName: \"kubernetes.io/projected/8808ab32-7df7-45ea-9abb-924ab62e8728-kube-api-access-2cx8c\") pod \"openshift-lws-operator-fd8ccff4c-hn6hg\" (UID: \"8808ab32-7df7-45ea-9abb-924ab62e8728\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.391111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.390937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpp88\" (UniqueName: \"kubernetes.io/projected/3aafbb19-454e-4ffe-9a6a-2ca0c66a649a-kube-api-access-tpp88\") pod \"cert-manager-55b4d954d5-vd6hn\" (UID: \"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a\") " pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.391111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.390967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3aafbb19-454e-4ffe-9a6a-2ca0c66a649a-bound-sa-token\") pod \"cert-manager-55b4d954d5-vd6hn\" (UID: \"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a\") " pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.391199 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.391179 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8808ab32-7df7-45ea-9abb-924ab62e8728-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hn6hg\" (UID: \"8808ab32-7df7-45ea-9abb-924ab62e8728\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.399452 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.399421 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cx8c\" (UniqueName: \"kubernetes.io/projected/8808ab32-7df7-45ea-9abb-924ab62e8728-kube-api-access-2cx8c\") pod \"openshift-lws-operator-fd8ccff4c-hn6hg\" (UID: \"8808ab32-7df7-45ea-9abb-924ab62e8728\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.449593 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.449558 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" Jul 08 19:09:18.491874 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.491838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpp88\" (UniqueName: \"kubernetes.io/projected/3aafbb19-454e-4ffe-9a6a-2ca0c66a649a-kube-api-access-tpp88\") pod \"cert-manager-55b4d954d5-vd6hn\" (UID: \"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a\") " pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.492047 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.491892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3aafbb19-454e-4ffe-9a6a-2ca0c66a649a-bound-sa-token\") pod \"cert-manager-55b4d954d5-vd6hn\" (UID: \"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a\") " pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.502165 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.502049 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpp88\" (UniqueName: \"kubernetes.io/projected/3aafbb19-454e-4ffe-9a6a-2ca0c66a649a-kube-api-access-tpp88\") pod \"cert-manager-55b4d954d5-vd6hn\" (UID: \"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a\") " pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.502740 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.502604 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3aafbb19-454e-4ffe-9a6a-2ca0c66a649a-bound-sa-token\") pod \"cert-manager-55b4d954d5-vd6hn\" (UID: \"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a\") " pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.533885 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.533856 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vd6hn" Jul 08 19:09:18.580632 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.580603 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg"] Jul 08 19:09:18.583175 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:09:18.583144 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8808ab32_7df7_45ea_9abb_924ab62e8728.slice/crio-5fb38f87c6c0343f6e1a79aba2ae380a7d2333eb9896550cf5a672fc43cccf95 WatchSource:0}: Error finding container 5fb38f87c6c0343f6e1a79aba2ae380a7d2333eb9896550cf5a672fc43cccf95: Status 404 returned error can't find the container with id 5fb38f87c6c0343f6e1a79aba2ae380a7d2333eb9896550cf5a672fc43cccf95 Jul 08 19:09:18.585246 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.585227 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:09:18.676408 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:18.676292 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vd6hn"] Jul 08 19:09:18.678960 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:09:18.678927 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3aafbb19_454e_4ffe_9a6a_2ca0c66a649a.slice/crio-dd0aab3913caf4677e5ab315d3afccdbe13b561559ff33ed111fb4b427b59931 WatchSource:0}: Error finding container dd0aab3913caf4677e5ab315d3afccdbe13b561559ff33ed111fb4b427b59931: Status 404 returned error can't find the container with id dd0aab3913caf4677e5ab315d3afccdbe13b561559ff33ed111fb4b427b59931 Jul 08 19:09:19.570935 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:19.570886 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vd6hn" event={"ID":"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a","Type":"ContainerStarted","Data":"dd0aab3913caf4677e5ab315d3afccdbe13b561559ff33ed111fb4b427b59931"} Jul 08 19:09:19.573581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:19.573497 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" event={"ID":"8808ab32-7df7-45ea-9abb-924ab62e8728","Type":"ContainerStarted","Data":"5fb38f87c6c0343f6e1a79aba2ae380a7d2333eb9896550cf5a672fc43cccf95"} Jul 08 19:09:23.593877 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:23.593785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" event={"ID":"8808ab32-7df7-45ea-9abb-924ab62e8728","Type":"ContainerStarted","Data":"766e315025a63a28c04c6b55505a87ee0ad241341927cca747b8e37644b62d9e"} Jul 08 19:09:23.595321 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:23.595295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vd6hn" event={"ID":"3aafbb19-454e-4ffe-9a6a-2ca0c66a649a","Type":"ContainerStarted","Data":"566a73a8ce676008c47022a459e679132592d978d72939f02eda0bacb95feeb2"} Jul 08 19:09:23.614483 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:23.614434 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hn6hg" podStartSLOduration=0.925543922 podStartE2EDuration="5.614416966s" totalImagesPullingTime="4.688873044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:23.612632854 +0000 UTC m=+309.051935487" watchObservedRunningTime="2026-07-08 19:09:23.614416966 +0000 UTC m=+309.053719596" Jul 08 19:09:23.634371 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:23.634323 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vd6hn" podStartSLOduration=1.03099505 podStartE2EDuration="5.634304897s" totalImagesPullingTime="4.603309847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:23.631670383 +0000 UTC m=+309.070973013" watchObservedRunningTime="2026-07-08 19:09:23.634304897 +0000 UTC m=+309.073607530" Jul 08 19:09:41.425940 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.425907 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49"] Jul 08 19:09:41.431861 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.431842 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.439997 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.439976 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 19:09:41.440118 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.440027 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 19:09:41.440118 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.440056 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 19:09:41.441596 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.441580 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 19:09:41.444824 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.444809 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dkxt2\"" Jul 08 19:09:41.453033 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.453009 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49"] Jul 08 19:09:41.484663 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.484637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799e5671-ff3e-407b-a3ce-c9268e717ae1-tmp\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.484766 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.484668 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822fq\" (UniqueName: \"kubernetes.io/projected/799e5671-ff3e-407b-a3ce-c9268e717ae1-kube-api-access-822fq\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.484766 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.484736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/799e5671-ff3e-407b-a3ce-c9268e717ae1-webhook-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.484840 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.484776 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/799e5671-ff3e-407b-a3ce-c9268e717ae1-manifests\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.484840 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.484835 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/799e5671-ff3e-407b-a3ce-c9268e717ae1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.585438 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.585400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/799e5671-ff3e-407b-a3ce-c9268e717ae1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.585438 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.585442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799e5671-ff3e-407b-a3ce-c9268e717ae1-tmp\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.585670 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.585468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822fq\" (UniqueName: \"kubernetes.io/projected/799e5671-ff3e-407b-a3ce-c9268e717ae1-kube-api-access-822fq\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.585670 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.585521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/799e5671-ff3e-407b-a3ce-c9268e717ae1-webhook-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.585781 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.585686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/799e5671-ff3e-407b-a3ce-c9268e717ae1-manifests\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.586059 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.586035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/799e5671-ff3e-407b-a3ce-c9268e717ae1-manifests\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.587778 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.587751 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799e5671-ff3e-407b-a3ce-c9268e717ae1-tmp\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.588043 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.588024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/799e5671-ff3e-407b-a3ce-c9268e717ae1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.588134 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.588096 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/799e5671-ff3e-407b-a3ce-c9268e717ae1-webhook-cert\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.596528 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.596509 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-822fq\" (UniqueName: \"kubernetes.io/projected/799e5671-ff3e-407b-a3ce-c9268e717ae1-kube-api-access-822fq\") pod \"opendatahub-operator-controller-manager-994cc9f69-84g49\" (UID: \"799e5671-ff3e-407b-a3ce-c9268e717ae1\") " pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.741641 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.741534 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:41.909699 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:41.909641 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49"] Jul 08 19:09:41.912032 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:09:41.912008 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799e5671_ff3e_407b_a3ce_c9268e717ae1.slice/crio-136601e4f872594c41443124e33eaf32c7220900069cdcb93ab79382fa990436 WatchSource:0}: Error finding container 136601e4f872594c41443124e33eaf32c7220900069cdcb93ab79382fa990436: Status 404 returned error can't find the container with id 136601e4f872594c41443124e33eaf32c7220900069cdcb93ab79382fa990436 Jul 08 19:09:42.678773 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:42.678734 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" event={"ID":"799e5671-ff3e-407b-a3ce-c9268e717ae1","Type":"ContainerStarted","Data":"136601e4f872594c41443124e33eaf32c7220900069cdcb93ab79382fa990436"} Jul 08 19:09:45.695312 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:45.695268 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" event={"ID":"799e5671-ff3e-407b-a3ce-c9268e717ae1","Type":"ContainerStarted","Data":"8f3ab32b16b434373e298bdabdbb097ff393c4a842354a781274d051c4896f96"} Jul 08 19:09:46.701738 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:46.701698 2569 generic.go:356] "Generic (PLEG): container finished" podID="799e5671-ff3e-407b-a3ce-c9268e717ae1" containerID="8f3ab32b16b434373e298bdabdbb097ff393c4a842354a781274d051c4896f96" exitCode=0 Jul 08 19:09:46.702107 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:46.701782 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" event={"ID":"799e5671-ff3e-407b-a3ce-c9268e717ae1","Type":"ContainerDied","Data":"8f3ab32b16b434373e298bdabdbb097ff393c4a842354a781274d051c4896f96"} Jul 08 19:09:47.708798 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:47.708758 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" event={"ID":"799e5671-ff3e-407b-a3ce-c9268e717ae1","Type":"ContainerStarted","Data":"830c641f4e464a016b34b64f6971a294be3a925f81bbf2d7df11c207b144f6ce"} Jul 08 19:09:47.709291 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:47.708829 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:47.733598 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:47.733534 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" podStartSLOduration=1.734366987 podStartE2EDuration="6.733509426s" totalImagesPullingTime="3.980273302s" totalInitContainerRuntime="1.018869137s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:47.732254561 +0000 UTC m=+333.171557218" watchObservedRunningTime="2026-07-08 19:09:47.733509426 +0000 UTC m=+333.172812057" Jul 08 19:09:58.716178 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.716149 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-994cc9f69-84g49" Jul 08 19:09:58.804984 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.804954 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw"] Jul 08 19:09:58.813371 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.813347 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:58.816356 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.816332 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 19:09:58.816696 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.816673 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 19:09:58.816796 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.816712 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v7zqq\"" Jul 08 19:09:58.816796 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.816732 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 19:09:58.821763 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.821739 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw"] Jul 08 19:09:58.946431 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.946387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-manager-config\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:58.946637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.946437 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-metrics-cert\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:58.946637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.946533 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-cert\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:58.946637 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:58.946625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbb2\" (UniqueName: \"kubernetes.io/projected/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-kube-api-access-rmbb2\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.047672 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.047592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbb2\" (UniqueName: \"kubernetes.io/projected/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-kube-api-access-rmbb2\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.047800 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.047683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-manager-config\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.047800 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.047721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-metrics-cert\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.047800 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.047776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-cert\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.048297 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.048261 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-manager-config\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.050218 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.050195 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-metrics-cert\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.050304 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.050292 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-cert\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.057311 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.057289 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbb2\" (UniqueName: \"kubernetes.io/projected/7b0398c0-28cd-4bbe-a107-6f9e33ad87d9-kube-api-access-rmbb2\") pod \"lws-controller-manager-69dfbfd578-x7lxw\" (UID: \"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9\") " pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.123797 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.123767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:09:59.279190 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.279163 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw"] Jul 08 19:09:59.281885 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:09:59.281857 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0398c0_28cd_4bbe_a107_6f9e33ad87d9.slice/crio-96cfc3d5853ba551dd535b83c5e7f6c83ceec56081002359c7e837f7b40605ae WatchSource:0}: Error finding container 96cfc3d5853ba551dd535b83c5e7f6c83ceec56081002359c7e837f7b40605ae: Status 404 returned error can't find the container with id 96cfc3d5853ba551dd535b83c5e7f6c83ceec56081002359c7e837f7b40605ae Jul 08 19:09:59.768045 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:09:59.768000 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" event={"ID":"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9","Type":"ContainerStarted","Data":"96cfc3d5853ba551dd535b83c5e7f6c83ceec56081002359c7e837f7b40605ae"} Jul 08 19:10:00.773994 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:00.773906 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" event={"ID":"7b0398c0-28cd-4bbe-a107-6f9e33ad87d9","Type":"ContainerStarted","Data":"f9802702deb7baf541f1ad900234a7a12ca95d6ff263bc4c9ea8bc27235158c7"} Jul 08 19:10:00.774354 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:00.774085 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:10:00.850747 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:00.850692 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" podStartSLOduration=1.613415804 podStartE2EDuration="2.850673536s" totalImagesPullingTime="1.237257732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:10:00.848501962 +0000 UTC m=+346.287804611" watchObservedRunningTime="2026-07-08 19:10:00.850673536 +0000 UTC m=+346.289976200" Jul 08 19:10:01.246650 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.246619 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdb786f-hkp72"] Jul 08 19:10:01.250178 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.250154 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.253448 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.253426 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 19:10:01.254078 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.254062 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 19:10:01.254132 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.254080 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2j6hl\"" Jul 08 19:10:01.254132 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.254094 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 19:10:01.254230 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.254092 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 19:10:01.270047 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.270013 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdb786f-hkp72"] Jul 08 19:10:01.371668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.371632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-tmp\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.371848 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.371682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-tls-certs\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.371848 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.371808 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf8l\" (UniqueName: \"kubernetes.io/projected/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-kube-api-access-zgf8l\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.472799 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.472760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf8l\" (UniqueName: \"kubernetes.io/projected/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-kube-api-access-zgf8l\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.472988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.472829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-tmp\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.472988 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.472876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-tls-certs\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.475154 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.475128 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-tmp\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.475435 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.475415 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-tls-certs\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.491117 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.491087 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf8l\" (UniqueName: \"kubernetes.io/projected/9f18b0e9-db82-489a-b3c7-723eaa8e5cb8-kube-api-access-zgf8l\") pod \"kube-auth-proxy-685cdb786f-hkp72\" (UID: \"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8\") " pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.559322 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.559237 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" Jul 08 19:10:01.710310 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.710278 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdb786f-hkp72"] Jul 08 19:10:01.711518 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:10:01.711493 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f18b0e9_db82_489a_b3c7_723eaa8e5cb8.slice/crio-f7219155139a3c05940d6969cda758da1a0597d8b73eee6018dd89fa179df428 WatchSource:0}: Error finding container f7219155139a3c05940d6969cda758da1a0597d8b73eee6018dd89fa179df428: Status 404 returned error can't find the container with id f7219155139a3c05940d6969cda758da1a0597d8b73eee6018dd89fa179df428 Jul 08 19:10:01.780514 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:01.780472 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" event={"ID":"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8","Type":"ContainerStarted","Data":"f7219155139a3c05940d6969cda758da1a0597d8b73eee6018dd89fa179df428"} Jul 08 19:10:06.811409 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:06.811359 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" event={"ID":"9f18b0e9-db82-489a-b3c7-723eaa8e5cb8","Type":"ContainerStarted","Data":"b6f084954e50bcab1a5082f0e03541ea381974bfe96c555b030bb769f16d6dca"} Jul 08 19:10:06.832509 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:06.832461 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-685cdb786f-hkp72" podStartSLOduration=1.327744136 podStartE2EDuration="5.832445941s" totalImagesPullingTime="4.504701805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:10:06.831402859 +0000 UTC m=+352.270705492" watchObservedRunningTime="2026-07-08 19:10:06.832445941 +0000 UTC m=+352.271748604" Jul 08 19:10:11.782686 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:10:11.782656 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69dfbfd578-x7lxw" Jul 08 19:11:43.888409 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.888333 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977"] Jul 08 19:11:43.891650 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.891634 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" Jul 08 19:11:43.898480 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.898461 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6tq27\"" Jul 08 19:11:43.898593 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.898486 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 19:11:43.898593 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.898497 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 19:11:43.898707 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.898691 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 19:11:43.917677 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.917655 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977"] Jul 08 19:11:43.983700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:43.983666 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hvq\" (UniqueName: \"kubernetes.io/projected/458ac938-570c-402b-b44c-fd57cb3ef0ae-kube-api-access-g8hvq\") pod \"dns-operator-controller-manager-864dd78c4c-xj977\" (UID: \"458ac938-570c-402b-b44c-fd57cb3ef0ae\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" Jul 08 19:11:44.084995 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.084958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hvq\" (UniqueName: \"kubernetes.io/projected/458ac938-570c-402b-b44c-fd57cb3ef0ae-kube-api-access-g8hvq\") pod \"dns-operator-controller-manager-864dd78c4c-xj977\" (UID: \"458ac938-570c-402b-b44c-fd57cb3ef0ae\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" Jul 08 19:11:44.101104 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.101071 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hvq\" (UniqueName: \"kubernetes.io/projected/458ac938-570c-402b-b44c-fd57cb3ef0ae-kube-api-access-g8hvq\") pod \"dns-operator-controller-manager-864dd78c4c-xj977\" (UID: \"458ac938-570c-402b-b44c-fd57cb3ef0ae\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" Jul 08 19:11:44.201668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.201633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" Jul 08 19:11:44.351977 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.351943 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977"] Jul 08 19:11:44.359690 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:11:44.359665 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458ac938_570c_402b_b44c_fd57cb3ef0ae.slice/crio-ba17ae1120d35e7e5992e86d25697720acda42bc1c8007964e8de7ea1d4182b9 WatchSource:0}: Error finding container ba17ae1120d35e7e5992e86d25697720acda42bc1c8007964e8de7ea1d4182b9: Status 404 returned error can't find the container with id ba17ae1120d35e7e5992e86d25697720acda42bc1c8007964e8de7ea1d4182b9 Jul 08 19:11:44.781846 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.781813 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-566b58fccd-47xhj"] Jul 08 19:11:44.785662 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.785645 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.799460 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.799434 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 19:11:44.799665 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.799495 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 19:11:44.799665 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.799494 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 19:11:44.801876 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.801858 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 19:11:44.802405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.802391 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tk955\"" Jul 08 19:11:44.808261 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.808236 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 19:11:44.841244 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.841209 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-566b58fccd-47xhj"] Jul 08 19:11:44.893457 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.893425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-trusted-ca-bundle\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.893929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.893533 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-serving-cert\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.893929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.893599 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-config\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.893929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.893652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-service-ca\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.893929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.893688 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-oauth-serving-cert\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.893929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.893856 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-oauth-config\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.893929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.893894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrw7m\" (UniqueName: \"kubernetes.io/projected/b6b22be7-2d22-44d6-88ff-512caa40c4d8-kube-api-access-zrw7m\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.910910 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.910887 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 19:11:44.995346 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.995313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrw7m\" (UniqueName: \"kubernetes.io/projected/b6b22be7-2d22-44d6-88ff-512caa40c4d8-kube-api-access-zrw7m\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.995510 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.995354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-trusted-ca-bundle\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.995510 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.995404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-serving-cert\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.995510 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.995421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-config\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.995510 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.995449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-service-ca\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.995510 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.995469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-oauth-serving-cert\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.995879 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.995573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-oauth-config\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.996422 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.996373 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-service-ca\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.996532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.996460 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-config\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.996532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.996473 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-trusted-ca-bundle\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.996777 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.996702 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b22be7-2d22-44d6-88ff-512caa40c4d8-oauth-serving-cert\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.999068 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.999043 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-oauth-config\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:44.999647 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:44.999620 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b22be7-2d22-44d6-88ff-512caa40c4d8-console-serving-cert\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:45.036959 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:45.036892 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrw7m\" (UniqueName: \"kubernetes.io/projected/b6b22be7-2d22-44d6-88ff-512caa40c4d8-kube-api-access-zrw7m\") pod \"console-566b58fccd-47xhj\" (UID: \"b6b22be7-2d22-44d6-88ff-512caa40c4d8\") " pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:45.095725 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:45.095400 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:45.254039 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:45.254005 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" event={"ID":"458ac938-570c-402b-b44c-fd57cb3ef0ae","Type":"ContainerStarted","Data":"ba17ae1120d35e7e5992e86d25697720acda42bc1c8007964e8de7ea1d4182b9"} Jul 08 19:11:45.594682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:45.594660 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-566b58fccd-47xhj"] Jul 08 19:11:45.596065 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:11:45.596034 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b22be7_2d22_44d6_88ff_512caa40c4d8.slice/crio-486e7610a1b1fb43ed09a7a589cdfe00c2c9ed81a6c17ff465a543784a661eb9 WatchSource:0}: Error finding container 486e7610a1b1fb43ed09a7a589cdfe00c2c9ed81a6c17ff465a543784a661eb9: Status 404 returned error can't find the container with id 486e7610a1b1fb43ed09a7a589cdfe00c2c9ed81a6c17ff465a543784a661eb9 Jul 08 19:11:46.259995 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:46.259964 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566b58fccd-47xhj" event={"ID":"b6b22be7-2d22-44d6-88ff-512caa40c4d8","Type":"ContainerStarted","Data":"ee7b852ce723e28bf7fc5afb1a128e9fdea3db0fc1e1da9d93e1d03b6ca5a7cd"} Jul 08 19:11:46.260308 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:46.260019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566b58fccd-47xhj" event={"ID":"b6b22be7-2d22-44d6-88ff-512caa40c4d8","Type":"ContainerStarted","Data":"486e7610a1b1fb43ed09a7a589cdfe00c2c9ed81a6c17ff465a543784a661eb9"} Jul 08 19:11:47.266127 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:47.266090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" event={"ID":"458ac938-570c-402b-b44c-fd57cb3ef0ae","Type":"ContainerStarted","Data":"97685f95eab8a13c36df3789fdb1a6321176aa21aefd409cb8e327943dbf4bb6"} Jul 08 19:11:47.266474 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:47.266316 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" Jul 08 19:11:47.297774 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:47.297727 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-566b58fccd-47xhj" podStartSLOduration=3.2977121560000002 podStartE2EDuration="3.297712156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:46.303969219 +0000 UTC m=+451.743271852" watchObservedRunningTime="2026-07-08 19:11:47.297712156 +0000 UTC m=+452.737014790" Jul 08 19:11:47.298692 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:47.298664 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" podStartSLOduration=2.400376203 podStartE2EDuration="4.298651589s" totalImagesPullingTime="1.898275386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:47.2972405 +0000 UTC m=+452.736543131" watchObservedRunningTime="2026-07-08 19:11:47.298651589 +0000 UTC m=+452.737954219" Jul 08 19:11:50.444651 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.444618 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq"] Jul 08 19:11:50.448058 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.448043 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.450256 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.450221 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 08 19:11:50.450384 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.450233 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 08 19:11:50.450471 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.450454 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2vkt6\"" Jul 08 19:11:50.456739 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.456718 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq"] Jul 08 19:11:50.543377 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.543345 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca9c2331-85e4-4f8b-919c-e677298dd07d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.543554 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.543419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca9c2331-85e4-4f8b-919c-e677298dd07d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.543554 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.543440 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mm9\" (UniqueName: \"kubernetes.io/projected/ca9c2331-85e4-4f8b-919c-e677298dd07d-kube-api-access-n9mm9\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.644875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.644840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca9c2331-85e4-4f8b-919c-e677298dd07d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.644875 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.644875 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mm9\" (UniqueName: \"kubernetes.io/projected/ca9c2331-85e4-4f8b-919c-e677298dd07d-kube-api-access-n9mm9\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.645102 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.644952 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca9c2331-85e4-4f8b-919c-e677298dd07d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.645645 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.645625 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca9c2331-85e4-4f8b-919c-e677298dd07d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.647359 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.647339 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca9c2331-85e4-4f8b-919c-e677298dd07d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.654168 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.654124 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mm9\" (UniqueName: \"kubernetes.io/projected/ca9c2331-85e4-4f8b-919c-e677298dd07d-kube-api-access-n9mm9\") pod \"kuadrant-console-plugin-66b5774985-9t9fq\" (UID: \"ca9c2331-85e4-4f8b-919c-e677298dd07d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.757710 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.757624 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" Jul 08 19:11:50.901126 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:50.901091 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq"] Jul 08 19:11:50.904010 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:11:50.903977 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca9c2331_85e4_4f8b_919c_e677298dd07d.slice/crio-a0ef7b57685a65d43be85abcd5ab270e5959da8a7c6cfea1abe5f52060a4f008 WatchSource:0}: Error finding container a0ef7b57685a65d43be85abcd5ab270e5959da8a7c6cfea1abe5f52060a4f008: Status 404 returned error can't find the container with id a0ef7b57685a65d43be85abcd5ab270e5959da8a7c6cfea1abe5f52060a4f008 Jul 08 19:11:51.286725 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:51.286687 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" event={"ID":"ca9c2331-85e4-4f8b-919c-e677298dd07d","Type":"ContainerStarted","Data":"a0ef7b57685a65d43be85abcd5ab270e5959da8a7c6cfea1abe5f52060a4f008"} Jul 08 19:11:55.102804 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:55.102775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:55.103318 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:55.103296 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:55.103455 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:55.103385 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:55.109148 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:55.109127 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-566b58fccd-47xhj" Jul 08 19:11:58.273592 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:11:58.273562 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xj977" Jul 08 19:12:15.418806 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:15.418772 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" event={"ID":"ca9c2331-85e4-4f8b-919c-e677298dd07d","Type":"ContainerStarted","Data":"f4b6b82a23aabf2875bfd57e03fe31fa12914a05f7b083c410a35332a7e12e7f"} Jul 08 19:12:37.776887 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.776828 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9t9fq" podStartSLOduration=24.019279349 podStartE2EDuration="47.776809251s" totalImagesPullingTime="23.757529902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:15.474244645 +0000 UTC m=+480.913547276" watchObservedRunningTime="2026-07-08 19:12:37.776809251 +0000 UTC m=+503.216111882" Jul 08 19:12:37.778037 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.778016 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:12:37.780462 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.780438 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:37.782934 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.782908 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 19:12:37.791979 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.791955 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:12:37.832461 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.832426 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:12:37.895447 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.895405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrzl\" (UniqueName: \"kubernetes.io/projected/600443f6-e855-48ab-b5a0-411d96f04088-kube-api-access-fxrzl\") pod \"limitador-limitador-54d6d988d5-wn8f4\" (UID: \"600443f6-e855-48ab-b5a0-411d96f04088\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:37.895641 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.895519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/600443f6-e855-48ab-b5a0-411d96f04088-config-file\") pod \"limitador-limitador-54d6d988d5-wn8f4\" (UID: \"600443f6-e855-48ab-b5a0-411d96f04088\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:37.996517 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.996484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/600443f6-e855-48ab-b5a0-411d96f04088-config-file\") pod \"limitador-limitador-54d6d988d5-wn8f4\" (UID: \"600443f6-e855-48ab-b5a0-411d96f04088\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:37.996674 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.996576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrzl\" (UniqueName: \"kubernetes.io/projected/600443f6-e855-48ab-b5a0-411d96f04088-kube-api-access-fxrzl\") pod \"limitador-limitador-54d6d988d5-wn8f4\" (UID: \"600443f6-e855-48ab-b5a0-411d96f04088\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:37.997111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:37.997090 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/600443f6-e855-48ab-b5a0-411d96f04088-config-file\") pod \"limitador-limitador-54d6d988d5-wn8f4\" (UID: \"600443f6-e855-48ab-b5a0-411d96f04088\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:38.005075 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:38.005049 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrzl\" (UniqueName: \"kubernetes.io/projected/600443f6-e855-48ab-b5a0-411d96f04088-kube-api-access-fxrzl\") pod \"limitador-limitador-54d6d988d5-wn8f4\" (UID: \"600443f6-e855-48ab-b5a0-411d96f04088\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:38.091322 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:38.091232 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:38.220965 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:38.220942 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:12:38.222884 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:12:38.222853 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600443f6_e855_48ab_b5a0_411d96f04088.slice/crio-98183fc91cd4faf0fcc31bae8bdb7d370360d5655f5366dc103ba64f83a3c31d WatchSource:0}: Error finding container 98183fc91cd4faf0fcc31bae8bdb7d370360d5655f5366dc103ba64f83a3c31d: Status 404 returned error can't find the container with id 98183fc91cd4faf0fcc31bae8bdb7d370360d5655f5366dc103ba64f83a3c31d Jul 08 19:12:38.531720 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:38.531683 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" event={"ID":"600443f6-e855-48ab-b5a0-411d96f04088","Type":"ContainerStarted","Data":"98183fc91cd4faf0fcc31bae8bdb7d370360d5655f5366dc103ba64f83a3c31d"} Jul 08 19:12:41.549591 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:41.549526 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" event={"ID":"600443f6-e855-48ab-b5a0-411d96f04088","Type":"ContainerStarted","Data":"a96a3a91930faa36ea7c86cd1169aa5b9c8af7f9e2a2ef662a0905b8ba71c2eb"} Jul 08 19:12:41.549965 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:41.549622 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:12:41.572924 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:41.572865 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" podStartSLOduration=1.803874955 podStartE2EDuration="4.572844937s" totalImagesPullingTime="2.768969982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:41.570260597 +0000 UTC m=+507.009563257" watchObservedRunningTime="2026-07-08 19:12:41.572844937 +0000 UTC m=+507.012147568" Jul 08 19:12:52.555519 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:12:52.555481 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wn8f4" Jul 08 19:13:54.528571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:13:54.528506 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:14:14.841090 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:14.841051 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:14:14.960335 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:14.960299 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:14:14.963698 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:14.963669 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:14:40.016676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:40.016593 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:14:44.400215 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.400172 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn"] Jul 08 19:14:44.403884 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.403860 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.406403 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.406372 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 19:14:44.407176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.407152 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 19:14:44.407293 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.407210 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 08 19:14:44.407293 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.407262 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7lzf5\"" Jul 08 19:14:44.414738 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.414717 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn"] Jul 08 19:14:44.464923 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.464891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.464923 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.464929 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.465114 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.464988 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.465114 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.465026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.465114 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.465072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.465114 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.465092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxq5w\" (UniqueName: \"kubernetes.io/projected/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-kube-api-access-jxq5w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566140 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566140 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566324 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566324 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxq5w\" (UniqueName: \"kubernetes.io/projected/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-kube-api-access-jxq5w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566324 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566324 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566692 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566656 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566692 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566685 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.566869 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.566727 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.568682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.568660 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.568928 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.568913 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.575895 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.575874 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxq5w\" (UniqueName: \"kubernetes.io/projected/7ff6b9f9-6174-4df9-a854-ebb0a4744d40-kube-api-access-jxq5w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5f4vn\" (UID: \"7ff6b9f9-6174-4df9-a854-ebb0a4744d40\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.715247 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.715196 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:44.854591 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.854565 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn"] Jul 08 19:14:44.856774 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:14:44.856744 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff6b9f9_6174_4df9_a854_ebb0a4744d40.slice/crio-c1e1c381c11140ebd47775d858a4a6e5fd54eca16872c058c558fc774cd61db7 WatchSource:0}: Error finding container c1e1c381c11140ebd47775d858a4a6e5fd54eca16872c058c558fc774cd61db7: Status 404 returned error can't find the container with id c1e1c381c11140ebd47775d858a4a6e5fd54eca16872c058c558fc774cd61db7 Jul 08 19:14:44.858252 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:44.858231 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:14:45.139939 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:45.139859 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" event={"ID":"7ff6b9f9-6174-4df9-a854-ebb0a4744d40","Type":"ContainerStarted","Data":"c1e1c381c11140ebd47775d858a4a6e5fd54eca16872c058c558fc774cd61db7"} Jul 08 19:14:45.724192 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:45.724155 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:14:46.145678 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:46.145577 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" event={"ID":"7ff6b9f9-6174-4df9-a854-ebb0a4744d40","Type":"ContainerStarted","Data":"f2b76bbcc4e6e09940ebb33b294480324e461c439e29b053decffb719a029a0e"} Jul 08 19:14:46.166726 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:46.166669 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" podStartSLOduration=1.1336764129999999 podStartE2EDuration="2.166654515s" totalImagesPullingTime="1.032978102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:14:46.165611434 +0000 UTC m=+631.604914066" watchObservedRunningTime="2026-07-08 19:14:46.166654515 +0000 UTC m=+631.605957145" Jul 08 19:14:51.716760 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:51.716725 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:14:54.715673 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:54.715618 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:54.716120 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:54.715714 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:54.728262 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:54.728238 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:55.199871 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:55.199844 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5f4vn" Jul 08 19:14:59.033162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:14:59.033129 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:15:01.826600 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.826561 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j"] Jul 08 19:15:01.833527 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.833496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:01.836434 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.836411 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 19:15:01.847848 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.847820 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j"] Jul 08 19:15:01.928344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.928308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:01.928344 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.928349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:01.928631 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.928374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjh6d\" (UniqueName: \"kubernetes.io/projected/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-kube-api-access-hjh6d\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:01.928631 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.928464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:01.928631 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.928510 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:01.928631 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:01.928561 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.029899 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.029856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.029899 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.029904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.030146 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.029935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjh6d\" (UniqueName: \"kubernetes.io/projected/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-kube-api-access-hjh6d\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.030146 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.029964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.030146 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.029999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.030146 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.030023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.030455 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.030425 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.030455 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.030441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.030654 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.030511 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.032322 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.032292 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.032498 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.032483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.039257 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.039238 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjh6d\" (UniqueName: \"kubernetes.io/projected/2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a-kube-api-access-hjh6d\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j\" (UID: \"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.143475 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.143377 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:02.284037 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:02.284013 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j"] Jul 08 19:15:02.286104 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:15:02.286077 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cf4ce8b_c15f_457b_8da8_653a7e8f3a5a.slice/crio-e28e6d505091befb8650655d2b809dfb6477ababc67953bc99f71373f5b8d52f WatchSource:0}: Error finding container e28e6d505091befb8650655d2b809dfb6477ababc67953bc99f71373f5b8d52f: Status 404 returned error can't find the container with id e28e6d505091befb8650655d2b809dfb6477ababc67953bc99f71373f5b8d52f Jul 08 19:15:03.227175 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:03.227139 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" event={"ID":"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a","Type":"ContainerStarted","Data":"ef50e1d5902851431bf9cff7756224890884344f7358507be4a1cb3d3e8688d1"} Jul 08 19:15:03.227584 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:03.227185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" event={"ID":"2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a","Type":"ContainerStarted","Data":"e28e6d505091befb8650655d2b809dfb6477ababc67953bc99f71373f5b8d52f"} Jul 08 19:15:03.248424 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:03.248370 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" podStartSLOduration=2.002459637 podStartE2EDuration="2.248354122s" totalImagesPullingTime="245.894485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:03.247410453 +0000 UTC m=+648.686713086" watchObservedRunningTime="2026-07-08 19:15:03.248354122 +0000 UTC m=+648.687656731" Jul 08 19:15:05.741293 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:05.741257 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:15:12.143963 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:12.143924 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:12.144440 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:12.143980 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:12.156318 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:12.156290 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:15:12.289929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:15:12.289893 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j" Jul 08 19:16:40.225217 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:16:40.225184 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:16:50.017178 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:16:50.017144 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:16:59.418329 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:16:59.418295 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:17:09.817904 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:17:09.817866 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:17:19.520485 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:17:19.520448 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:17:48.610849 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:17:48.610776 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:18:00.853510 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:18:00.853476 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:18:34.608677 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:18:34.608644 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:18:42.705379 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:18:42.705340 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:18:53.844924 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:18:53.844893 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:19:02.022633 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:19:02.022588 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:19:12.408829 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:19:12.408742 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:19:14.996682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:19:14.996652 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:19:14.997147 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:19:14.996783 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:19:24.728265 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:19:24.728236 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:19:35.415259 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:19:35.415215 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:19:45.713004 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:19:45.712968 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:20:08.008793 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:20:08.008752 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:20:19.518220 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:20:19.518188 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:20:43.230068 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:20:43.229991 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:20:55.531411 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:20:55.531375 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:21:10.907460 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:21:10.907423 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:21:20.236120 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:21:20.236080 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:21:30.323220 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:21:30.323186 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:21:40.104323 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:21:40.104280 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:21:49.321087 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:21:49.321054 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:23:40.310762 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:23:40.310677 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:23:50.210814 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:23:50.210775 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:24:03.827661 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:03.827627 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:24:13.112825 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:13.112793 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:24:15.026953 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:15.026923 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:24:15.027839 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:15.027819 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:24:22.908130 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:22.908095 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:24:31.715177 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:31.715138 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:24:43.410156 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:43.410122 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:24:52.411084 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:24:52.411050 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:25:02.609917 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:25:02.609877 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:25:12.310083 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:25:12.309997 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:25:20.303943 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:25:20.303904 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:25:29.509825 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:25:29.509788 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:25:52.312984 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:25:52.312942 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:26:02.512187 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:26:02.512147 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:26:12.020881 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:26:12.020836 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:26:21.912430 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:26:21.912390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:26:40.008652 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:26:40.008574 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:26:55.129398 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:26:55.129364 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:27:04.620956 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:27:04.620914 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:27:18.213390 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:27:18.213349 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:27:27.813531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:27:27.813497 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:27:41.508406 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:27:41.508371 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:27:50.800662 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:27:50.800633 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:28:00.217963 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:28:00.217928 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:28:13.414854 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:28:13.414773 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:28:22.814206 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:28:22.814167 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:28:29.707567 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:28:29.707517 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:28:44.508788 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:28:44.508753 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:28:57.410676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:28:57.410642 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:29:06.515611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:29:06.515576 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:29:13.855194 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:29:13.855148 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:29:15.056767 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:29:15.056737 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:29:15.058920 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:29:15.058896 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:29:22.909747 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:29:22.909712 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:29:35.405585 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:29:35.405480 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:29:44.902884 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:29:44.902845 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:30:06.506376 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:30:06.506335 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:30:18.525043 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:30:18.525003 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:31:55.115962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:31:55.115922 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:32:04.213507 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:32:04.213474 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:32:14.503792 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:32:14.503742 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:32:28.114166 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:32:28.114127 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:32:59.864051 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:32:59.863978 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:33:18.207284 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:33:18.207245 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:33:33.415615 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:33:33.415568 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:33:46.706318 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:33:46.706283 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:34:15.087531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:34:15.087504 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:34:15.090091 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:34:15.090071 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:34:26.411784 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:34:26.411739 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:34:35.613247 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:34:35.613209 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:34:49.010499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:34:49.010463 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:35:33.625581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:33.625530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:35:48.205576 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:48.205482 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:35:58.100906 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.100869 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f"] Jul 08 19:35:58.104420 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.104401 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.106828 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.106804 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8ab38c\"/\"kube-root-ca.crt\"" Jul 08 19:35:58.106952 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.106858 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8ab38c\"/\"default-dockercfg-pv98z\"" Jul 08 19:35:58.107723 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.107702 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8ab38c\"/\"test-model-8ab38c-kserve-self-signed-certs\"" Jul 08 19:35:58.107794 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.107731 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8ab38c\"/\"openshift-service-ca.crt\"" Jul 08 19:35:58.118353 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.118327 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f"] Jul 08 19:35:58.238475 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.238434 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-tmp-dir\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.238658 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.238504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-dshm\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.238658 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.238524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-home\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.238658 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.238624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhz8p\" (UniqueName: \"kubernetes.io/projected/75fb3397-0bdd-4aed-96be-b1a0f969a447-kube-api-access-zhz8p\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.238765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.238657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-model-cache\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.238765 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.238700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fb3397-0bdd-4aed-96be-b1a0f969a447-tls-certs\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.339826 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.339792 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fb3397-0bdd-4aed-96be-b1a0f969a447-tls-certs\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.339987 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.339844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-tmp-dir\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.339987 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.339885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-dshm\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.339987 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.339907 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-home\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.339987 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.339962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhz8p\" (UniqueName: \"kubernetes.io/projected/75fb3397-0bdd-4aed-96be-b1a0f969a447-kube-api-access-zhz8p\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.340179 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.340012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-model-cache\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.340401 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.340383 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-tmp-dir\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.340672 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.340416 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-model-cache\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.340672 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.340467 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-home\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.342198 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.342171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-dshm\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.342334 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.342318 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fb3397-0bdd-4aed-96be-b1a0f969a447-tls-certs\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.348660 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.348633 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhz8p\" (UniqueName: \"kubernetes.io/projected/75fb3397-0bdd-4aed-96be-b1a0f969a447-kube-api-access-zhz8p\") pod \"test-model-8ab38c-kserve-57bcdf9787-8sq8f\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.414647 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.414606 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:35:58.551738 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:58.551715 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f"] Jul 08 19:35:58.553880 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:35:58.553837 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75fb3397_0bdd_4aed_96be_b1a0f969a447.slice/crio-29f7778ca4ecdc1011db52b90f65b3ba9f4cd2c51cd611c8c8c0d04f9229e273 WatchSource:0}: Error finding container 29f7778ca4ecdc1011db52b90f65b3ba9f4cd2c51cd611c8c8c0d04f9229e273: Status 404 returned error can't find the container with id 29f7778ca4ecdc1011db52b90f65b3ba9f4cd2c51cd611c8c8c0d04f9229e273 Jul 08 19:35:59.319240 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:59.319206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" event={"ID":"75fb3397-0bdd-4aed-96be-b1a0f969a447","Type":"ContainerStarted","Data":"b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373"} Jul 08 19:35:59.319240 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:59.319243 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" event={"ID":"75fb3397-0bdd-4aed-96be-b1a0f969a447","Type":"ContainerStarted","Data":"29f7778ca4ecdc1011db52b90f65b3ba9f4cd2c51cd611c8c8c0d04f9229e273"} Jul 08 19:35:59.342474 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:35:59.342415 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" podStartSLOduration=1.342392751 podStartE2EDuration="1.342392751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:35:59.341046874 +0000 UTC m=+1904.780349504" watchObservedRunningTime="2026-07-08 19:35:59.342392751 +0000 UTC m=+1904.781695383" Jul 08 19:36:02.013605 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:02.013566 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:36:08.415707 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:08.415674 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:36:08.416106 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:08.415719 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:36:08.428159 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:08.428133 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:36:09.386484 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:09.386455 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:36:13.321791 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:13.321761 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:36:28.029307 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.029273 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw"] Jul 08 19:36:28.032774 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.032757 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.035960 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.035939 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1709ef\"/\"kube-root-ca.crt\"" Jul 08 19:36:28.035960 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.035953 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1709ef\"/\"test-model-1709ef-kserve-self-signed-certs\"" Jul 08 19:36:28.036164 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.035972 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1709ef\"/\"openshift-service-ca.crt\"" Jul 08 19:36:28.036164 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.035942 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1709ef\"/\"default-dockercfg-9bbtz\"" Jul 08 19:36:28.042589 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.042564 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw"] Jul 08 19:36:28.125236 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.125193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8njd\" (UniqueName: \"kubernetes.io/projected/fd235791-a70a-4383-978d-b6ecdc0d2bed-kube-api-access-c8njd\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.125426 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.125250 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd235791-a70a-4383-978d-b6ecdc0d2bed-tls-certs\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.125426 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.125307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-dshm\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.125426 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.125327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-model-cache\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.125426 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.125361 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-tmp-dir\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.125426 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.125384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-home\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226446 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226410 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-dshm\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226446 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-model-cache\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226691 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-tmp-dir\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226691 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226509 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-home\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226691 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8njd\" (UniqueName: \"kubernetes.io/projected/fd235791-a70a-4383-978d-b6ecdc0d2bed-kube-api-access-c8njd\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226691 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226649 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd235791-a70a-4383-978d-b6ecdc0d2bed-tls-certs\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226917 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226894 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-model-cache\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226985 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226926 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-tmp-dir\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.226985 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.226971 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-home\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.228793 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.228774 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-dshm\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.229154 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.229135 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd235791-a70a-4383-978d-b6ecdc0d2bed-tls-certs\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.234290 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.234269 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8njd\" (UniqueName: \"kubernetes.io/projected/fd235791-a70a-4383-978d-b6ecdc0d2bed-kube-api-access-c8njd\") pod \"test-model-1709ef-kserve-6995d69c89-mngnw\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.342976 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.342892 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:28.484141 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:28.484118 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw"] Jul 08 19:36:28.486376 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:36:28.486342 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd235791_a70a_4383_978d_b6ecdc0d2bed.slice/crio-80c71fac6f28b0d89358790ed9f9dbde2771d677497530ede4b0fd589896f59b WatchSource:0}: Error finding container 80c71fac6f28b0d89358790ed9f9dbde2771d677497530ede4b0fd589896f59b: Status 404 returned error can't find the container with id 80c71fac6f28b0d89358790ed9f9dbde2771d677497530ede4b0fd589896f59b Jul 08 19:36:29.477000 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:29.476960 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" event={"ID":"fd235791-a70a-4383-978d-b6ecdc0d2bed","Type":"ContainerStarted","Data":"f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e"} Jul 08 19:36:29.477000 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:29.477003 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" event={"ID":"fd235791-a70a-4383-978d-b6ecdc0d2bed","Type":"ContainerStarted","Data":"80c71fac6f28b0d89358790ed9f9dbde2771d677497530ede4b0fd589896f59b"} Jul 08 19:36:29.497581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:29.497502 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" podStartSLOduration=1.4974868 podStartE2EDuration="1.4974868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:36:29.495814472 +0000 UTC m=+1934.935117114" watchObservedRunningTime="2026-07-08 19:36:29.4974868 +0000 UTC m=+1934.936789431" Jul 08 19:36:33.518643 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:33.518600 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:36:38.343848 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:38.343806 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:38.344219 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:38.343875 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:38.356338 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:38.356316 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:38.532126 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:38.532097 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:36:56.233343 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:56.233309 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f"] Jul 08 19:36:57.183975 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:36:57.183945 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw"] Jul 08 19:37:01.011684 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:01.011642 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:37:11.254178 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.254064 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" podUID="75fb3397-0bdd-4aed-96be-b1a0f969a447" containerName="main" containerID="cri-o://b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373" gracePeriod=45 Jul 08 19:37:11.511834 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.511772 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:37:11.629106 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629066 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-tmp-dir\") pod \"75fb3397-0bdd-4aed-96be-b1a0f969a447\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " Jul 08 19:37:11.629106 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629107 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-home\") pod \"75fb3397-0bdd-4aed-96be-b1a0f969a447\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " Jul 08 19:37:11.629370 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629166 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhz8p\" (UniqueName: \"kubernetes.io/projected/75fb3397-0bdd-4aed-96be-b1a0f969a447-kube-api-access-zhz8p\") pod \"75fb3397-0bdd-4aed-96be-b1a0f969a447\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " Jul 08 19:37:11.629370 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629197 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-dshm\") pod \"75fb3397-0bdd-4aed-96be-b1a0f969a447\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " Jul 08 19:37:11.629370 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629229 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fb3397-0bdd-4aed-96be-b1a0f969a447-tls-certs\") pod \"75fb3397-0bdd-4aed-96be-b1a0f969a447\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " Jul 08 19:37:11.629370 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629249 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-model-cache\") pod \"75fb3397-0bdd-4aed-96be-b1a0f969a447\" (UID: \"75fb3397-0bdd-4aed-96be-b1a0f969a447\") " Jul 08 19:37:11.629612 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629415 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-home" (OuterVolumeSpecName: "home") pod "75fb3397-0bdd-4aed-96be-b1a0f969a447" (UID: "75fb3397-0bdd-4aed-96be-b1a0f969a447"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:11.629612 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629425 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75fb3397-0bdd-4aed-96be-b1a0f969a447" (UID: "75fb3397-0bdd-4aed-96be-b1a0f969a447"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:11.629714 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629620 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-model-cache" (OuterVolumeSpecName: "model-cache") pod "75fb3397-0bdd-4aed-96be-b1a0f969a447" (UID: "75fb3397-0bdd-4aed-96be-b1a0f969a447"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:11.629714 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629653 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-tmp-dir\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:11.629714 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.629673 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-home\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:11.631611 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.631586 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75fb3397-0bdd-4aed-96be-b1a0f969a447-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75fb3397-0bdd-4aed-96be-b1a0f969a447" (UID: "75fb3397-0bdd-4aed-96be-b1a0f969a447"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:37:11.631704 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.631635 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fb3397-0bdd-4aed-96be-b1a0f969a447-kube-api-access-zhz8p" (OuterVolumeSpecName: "kube-api-access-zhz8p") pod "75fb3397-0bdd-4aed-96be-b1a0f969a447" (UID: "75fb3397-0bdd-4aed-96be-b1a0f969a447"). InnerVolumeSpecName "kube-api-access-zhz8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:37:11.631704 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.631657 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-dshm" (OuterVolumeSpecName: "dshm") pod "75fb3397-0bdd-4aed-96be-b1a0f969a447" (UID: "75fb3397-0bdd-4aed-96be-b1a0f969a447"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:11.682453 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.682409 2569 generic.go:356] "Generic (PLEG): container finished" podID="75fb3397-0bdd-4aed-96be-b1a0f969a447" containerID="b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373" exitCode=0 Jul 08 19:37:11.682676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.682458 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" Jul 08 19:37:11.682676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.682499 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" event={"ID":"75fb3397-0bdd-4aed-96be-b1a0f969a447","Type":"ContainerDied","Data":"b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373"} Jul 08 19:37:11.682676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.682573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f" event={"ID":"75fb3397-0bdd-4aed-96be-b1a0f969a447","Type":"ContainerDied","Data":"29f7778ca4ecdc1011db52b90f65b3ba9f4cd2c51cd611c8c8c0d04f9229e273"} Jul 08 19:37:11.682676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.682597 2569 scope.go:117] "RemoveContainer" containerID="b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373" Jul 08 19:37:11.692012 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.691994 2569 scope.go:117] "RemoveContainer" containerID="b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373" Jul 08 19:37:11.692280 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:37:11.692259 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373\": container with ID starting with b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373 not found: ID does not exist" containerID="b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373" Jul 08 19:37:11.692327 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.692291 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373"} err="failed to get container status \"b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373\": rpc error: code = NotFound desc = could not find container \"b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373\": container with ID starting with b66f052326b7f77aae9c9a6d4d9c2bec6641d969f3def1c380ae53b7b1c24373 not found: ID does not exist" Jul 08 19:37:11.703894 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.703869 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f"] Jul 08 19:37:11.705557 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.705517 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8ab38c/test-model-8ab38c-kserve-57bcdf9787-8sq8f"] Jul 08 19:37:11.731022 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.730987 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhz8p\" (UniqueName: \"kubernetes.io/projected/75fb3397-0bdd-4aed-96be-b1a0f969a447-kube-api-access-zhz8p\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:11.731022 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.731020 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-dshm\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:11.731163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.731030 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fb3397-0bdd-4aed-96be-b1a0f969a447-tls-certs\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:11.731163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:11.731039 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fb3397-0bdd-4aed-96be-b1a0f969a447-model-cache\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:12.204598 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.204519 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" podUID="fd235791-a70a-4383-978d-b6ecdc0d2bed" containerName="main" containerID="cri-o://f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e" gracePeriod=45 Jul 08 19:37:12.463915 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.463850 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:37:12.538057 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8njd\" (UniqueName: \"kubernetes.io/projected/fd235791-a70a-4383-978d-b6ecdc0d2bed-kube-api-access-c8njd\") pod \"fd235791-a70a-4383-978d-b6ecdc0d2bed\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " Jul 08 19:37:12.538231 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538109 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-dshm\") pod \"fd235791-a70a-4383-978d-b6ecdc0d2bed\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " Jul 08 19:37:12.538231 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538199 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-tmp-dir\") pod \"fd235791-a70a-4383-978d-b6ecdc0d2bed\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " Jul 08 19:37:12.538231 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538228 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-home\") pod \"fd235791-a70a-4383-978d-b6ecdc0d2bed\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " Jul 08 19:37:12.538374 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538267 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-model-cache\") pod \"fd235791-a70a-4383-978d-b6ecdc0d2bed\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " Jul 08 19:37:12.538374 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538302 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd235791-a70a-4383-978d-b6ecdc0d2bed-tls-certs\") pod \"fd235791-a70a-4383-978d-b6ecdc0d2bed\" (UID: \"fd235791-a70a-4383-978d-b6ecdc0d2bed\") " Jul 08 19:37:12.538499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538449 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd235791-a70a-4383-978d-b6ecdc0d2bed" (UID: "fd235791-a70a-4383-978d-b6ecdc0d2bed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:12.538499 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538483 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-home" (OuterVolumeSpecName: "home") pod "fd235791-a70a-4383-978d-b6ecdc0d2bed" (UID: "fd235791-a70a-4383-978d-b6ecdc0d2bed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:12.538718 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538613 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd235791-a70a-4383-978d-b6ecdc0d2bed" (UID: "fd235791-a70a-4383-978d-b6ecdc0d2bed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:12.538777 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538726 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-tmp-dir\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:12.538777 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538746 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-home\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:12.538777 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.538762 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-model-cache\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:12.540422 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.540399 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-dshm" (OuterVolumeSpecName: "dshm") pod "fd235791-a70a-4383-978d-b6ecdc0d2bed" (UID: "fd235791-a70a-4383-978d-b6ecdc0d2bed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:37:12.540526 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.540425 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd235791-a70a-4383-978d-b6ecdc0d2bed-kube-api-access-c8njd" (OuterVolumeSpecName: "kube-api-access-c8njd") pod "fd235791-a70a-4383-978d-b6ecdc0d2bed" (UID: "fd235791-a70a-4383-978d-b6ecdc0d2bed"). InnerVolumeSpecName "kube-api-access-c8njd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:37:12.540656 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.540641 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd235791-a70a-4383-978d-b6ecdc0d2bed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd235791-a70a-4383-978d-b6ecdc0d2bed" (UID: "fd235791-a70a-4383-978d-b6ecdc0d2bed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:37:12.639285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.639249 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8njd\" (UniqueName: \"kubernetes.io/projected/fd235791-a70a-4383-978d-b6ecdc0d2bed-kube-api-access-c8njd\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:12.639285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.639279 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd235791-a70a-4383-978d-b6ecdc0d2bed-dshm\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:12.639285 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.639288 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd235791-a70a-4383-978d-b6ecdc0d2bed-tls-certs\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:37:12.688578 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.688526 2569 generic.go:356] "Generic (PLEG): container finished" podID="fd235791-a70a-4383-978d-b6ecdc0d2bed" containerID="f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e" exitCode=0 Jul 08 19:37:12.688578 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.688575 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" event={"ID":"fd235791-a70a-4383-978d-b6ecdc0d2bed","Type":"ContainerDied","Data":"f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e"} Jul 08 19:37:12.688794 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.688587 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" Jul 08 19:37:12.688794 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.688619 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw" event={"ID":"fd235791-a70a-4383-978d-b6ecdc0d2bed","Type":"ContainerDied","Data":"80c71fac6f28b0d89358790ed9f9dbde2771d677497530ede4b0fd589896f59b"} Jul 08 19:37:12.688794 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.688637 2569 scope.go:117] "RemoveContainer" containerID="f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e" Jul 08 19:37:12.697625 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.697608 2569 scope.go:117] "RemoveContainer" containerID="f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e" Jul 08 19:37:12.697930 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:37:12.697913 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e\": container with ID starting with f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e not found: ID does not exist" containerID="f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e" Jul 08 19:37:12.697992 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.697936 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e"} err="failed to get container status \"f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e\": rpc error: code = NotFound desc = could not find container \"f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e\": container with ID starting with f806ed7c2113086f35dc2a9b47f5a37fee4627198f7aee9dce125ce2553fda8e not found: ID does not exist" Jul 08 19:37:12.711701 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.711680 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw"] Jul 08 19:37:12.715415 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:12.715370 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1709ef/test-model-1709ef-kserve-6995d69c89-mngnw"] Jul 08 19:37:13.099695 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:13.099627 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75fb3397-0bdd-4aed-96be-b1a0f969a447" path="/var/lib/kubelet/pods/75fb3397-0bdd-4aed-96be-b1a0f969a447/volumes" Jul 08 19:37:13.099991 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:13.099979 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd235791-a70a-4383-978d-b6ecdc0d2bed" path="/var/lib/kubelet/pods/fd235791-a70a-4383-978d-b6ecdc0d2bed/volumes" Jul 08 19:37:43.318711 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:43.318672 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:37:58.619032 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:37:58.618999 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:38:16.122181 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:38:16.122150 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:38:30.911871 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:38:30.911836 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:38:41.918351 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:38:41.918275 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:39:04.806289 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.806248 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn"] Jul 08 19:39:04.806945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.806925 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd235791-a70a-4383-978d-b6ecdc0d2bed" containerName="main" Jul 08 19:39:04.807040 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.806949 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd235791-a70a-4383-978d-b6ecdc0d2bed" containerName="main" Jul 08 19:39:04.807040 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.806971 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75fb3397-0bdd-4aed-96be-b1a0f969a447" containerName="main" Jul 08 19:39:04.807040 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.806980 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="75fb3397-0bdd-4aed-96be-b1a0f969a447" containerName="main" Jul 08 19:39:04.807196 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.807102 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="75fb3397-0bdd-4aed-96be-b1a0f969a447" containerName="main" Jul 08 19:39:04.807196 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.807117 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd235791-a70a-4383-978d-b6ecdc0d2bed" containerName="main" Jul 08 19:39:04.810401 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.810379 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.812828 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.812810 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d86987\"/\"kube-root-ca.crt\"" Jul 08 19:39:04.813659 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.813640 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d86987\"/\"openshift-service-ca.crt\"" Jul 08 19:39:04.813769 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.813643 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d86987\"/\"auth-test-model-d86987-kserve-self-signed-certs\"" Jul 08 19:39:04.813769 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.813702 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d86987\"/\"default-dockercfg-pbqfh\"" Jul 08 19:39:04.821772 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.821750 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn"] Jul 08 19:39:04.884623 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.884590 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-tmp-dir\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.884785 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.884644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-model-cache\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.884785 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.884674 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23238b6a-d33e-46b1-89b2-2fb0752debdb-tls-certs\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.884785 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.884697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-home\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.884785 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.884725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbxm\" (UniqueName: \"kubernetes.io/projected/23238b6a-d33e-46b1-89b2-2fb0752debdb-kube-api-access-dwbxm\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.884920 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.884814 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-dshm\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986276 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-model-cache\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986276 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23238b6a-d33e-46b1-89b2-2fb0752debdb-tls-certs\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986521 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-home\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986521 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbxm\" (UniqueName: \"kubernetes.io/projected/23238b6a-d33e-46b1-89b2-2fb0752debdb-kube-api-access-dwbxm\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986521 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-dshm\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986521 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-tmp-dir\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986761 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986738 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-model-cache\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986813 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986777 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-home\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.986854 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.986835 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-tmp-dir\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.988682 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.988663 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-dshm\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.988929 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.988912 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23238b6a-d33e-46b1-89b2-2fb0752debdb-tls-certs\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:04.994316 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:04.994294 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbxm\" (UniqueName: \"kubernetes.io/projected/23238b6a-d33e-46b1-89b2-2fb0752debdb-kube-api-access-dwbxm\") pod \"auth-test-model-d86987-kserve-76769d6c95-ddnmn\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:05.122100 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:05.122020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:05.264262 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:05.264222 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn"] Jul 08 19:39:05.266597 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:39:05.266573 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23238b6a_d33e_46b1_89b2_2fb0752debdb.slice/crio-ed5dd276321e8d7bed3f6a889c1de784d5cd34e3ced226348395c657ba5fbf49 WatchSource:0}: Error finding container ed5dd276321e8d7bed3f6a889c1de784d5cd34e3ced226348395c657ba5fbf49: Status 404 returned error can't find the container with id ed5dd276321e8d7bed3f6a889c1de784d5cd34e3ced226348395c657ba5fbf49 Jul 08 19:39:06.247912 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:06.247877 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" event={"ID":"23238b6a-d33e-46b1-89b2-2fb0752debdb","Type":"ContainerStarted","Data":"eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8"} Jul 08 19:39:06.247912 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:06.247917 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" event={"ID":"23238b6a-d33e-46b1-89b2-2fb0752debdb","Type":"ContainerStarted","Data":"ed5dd276321e8d7bed3f6a889c1de784d5cd34e3ced226348395c657ba5fbf49"} Jul 08 19:39:06.266045 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:06.265996 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" podStartSLOduration=2.26597473 podStartE2EDuration="2.26597473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:39:06.263626715 +0000 UTC m=+2091.702929358" watchObservedRunningTime="2026-07-08 19:39:06.26597473 +0000 UTC m=+2091.705277362" Jul 08 19:39:08.308163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:08.308131 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:39:15.121849 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:15.121814 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:39:15.122291 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:15.122199 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:15.122291 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:15.122243 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:15.124419 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:15.124396 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:39:15.135833 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:15.135814 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:15.301862 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:15.301831 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:39:26.414496 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:26.414459 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:39:41.016635 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:39:41.016602 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:40:00.115748 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:40:00.115715 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:40:16.557132 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:40:16.557097 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:40:35.424263 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:40:35.424226 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:40:54.102616 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:40:54.102581 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:41:12.303429 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:12.303394 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:41:25.028624 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:25.028590 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn"] Jul 08 19:41:28.002589 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:28.002550 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:41:33.608511 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.608477 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt"] Jul 08 19:41:33.612204 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.612186 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.614447 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.614426 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d86987\"/\"sub-test-model-d86987-kserve-self-signed-certs\"" Jul 08 19:41:33.620275 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.620254 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt"] Jul 08 19:41:33.723011 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.722974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883ba18f-4bba-436e-979f-16e634f99fa4-tls-certs\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.723205 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.723037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-dshm\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.723205 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.723084 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-model-cache\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.723205 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.723112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tk9h\" (UniqueName: \"kubernetes.io/projected/883ba18f-4bba-436e-979f-16e634f99fa4-kube-api-access-8tk9h\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.723205 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.723191 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-home\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.723350 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.723235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-tmp-dir\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.824706 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.824661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883ba18f-4bba-436e-979f-16e634f99fa4-tls-certs\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.824886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.824722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-dshm\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.824886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.824744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-model-cache\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.824886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.824761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tk9h\" (UniqueName: \"kubernetes.io/projected/883ba18f-4bba-436e-979f-16e634f99fa4-kube-api-access-8tk9h\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.824886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.824795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-home\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.824886 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.824824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-tmp-dir\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.825214 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.825191 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-tmp-dir\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.825289 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.825249 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-model-cache\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.825289 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.825262 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-home\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.827180 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.827152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-dshm\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.827309 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.827288 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883ba18f-4bba-436e-979f-16e634f99fa4-tls-certs\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.833581 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.833534 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tk9h\" (UniqueName: \"kubernetes.io/projected/883ba18f-4bba-436e-979f-16e634f99fa4-kube-api-access-8tk9h\") pod \"sub-test-model-d86987-kserve-5b87c87c5-qrtzt\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:33.921972 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:33.921946 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:34.053604 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:34.053580 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt"] Jul 08 19:41:34.055737 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:41:34.055705 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod883ba18f_4bba_436e_979f_16e634f99fa4.slice/crio-1bd47f667931ecfe7ab36fe328835f9a5f4e63a577c6997da14e6b1826e01f1c WatchSource:0}: Error finding container 1bd47f667931ecfe7ab36fe328835f9a5f4e63a577c6997da14e6b1826e01f1c: Status 404 returned error can't find the container with id 1bd47f667931ecfe7ab36fe328835f9a5f4e63a577c6997da14e6b1826e01f1c Jul 08 19:41:34.972325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:34.972251 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" event={"ID":"883ba18f-4bba-436e-979f-16e634f99fa4","Type":"ContainerStarted","Data":"09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68"} Jul 08 19:41:34.972325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:34.972289 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" event={"ID":"883ba18f-4bba-436e-979f-16e634f99fa4","Type":"ContainerStarted","Data":"1bd47f667931ecfe7ab36fe328835f9a5f4e63a577c6997da14e6b1826e01f1c"} Jul 08 19:41:34.990070 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:34.990018 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" podStartSLOduration=1.990002506 podStartE2EDuration="1.990002506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:41:34.988384248 +0000 UTC m=+2240.427686877" watchObservedRunningTime="2026-07-08 19:41:34.990002506 +0000 UTC m=+2240.429305137" Jul 08 19:41:40.049114 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.049054 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" podUID="23238b6a-d33e-46b1-89b2-2fb0752debdb" containerName="main" containerID="cri-o://eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8" gracePeriod=45 Jul 08 19:41:40.296102 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.296072 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:41:40.388730 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.388648 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-model-cache\") pod \"23238b6a-d33e-46b1-89b2-2fb0752debdb\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " Jul 08 19:41:40.388730 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.388694 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-tmp-dir\") pod \"23238b6a-d33e-46b1-89b2-2fb0752debdb\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " Jul 08 19:41:40.388954 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.388734 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-home\") pod \"23238b6a-d33e-46b1-89b2-2fb0752debdb\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " Jul 08 19:41:40.388954 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.388836 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-dshm\") pod \"23238b6a-d33e-46b1-89b2-2fb0752debdb\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " Jul 08 19:41:40.388954 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.388869 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23238b6a-d33e-46b1-89b2-2fb0752debdb-tls-certs\") pod \"23238b6a-d33e-46b1-89b2-2fb0752debdb\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " Jul 08 19:41:40.388954 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.388894 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwbxm\" (UniqueName: \"kubernetes.io/projected/23238b6a-d33e-46b1-89b2-2fb0752debdb-kube-api-access-dwbxm\") pod \"23238b6a-d33e-46b1-89b2-2fb0752debdb\" (UID: \"23238b6a-d33e-46b1-89b2-2fb0752debdb\") " Jul 08 19:41:40.389171 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.388952 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-model-cache" (OuterVolumeSpecName: "model-cache") pod "23238b6a-d33e-46b1-89b2-2fb0752debdb" (UID: "23238b6a-d33e-46b1-89b2-2fb0752debdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:41:40.389171 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.389030 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23238b6a-d33e-46b1-89b2-2fb0752debdb" (UID: "23238b6a-d33e-46b1-89b2-2fb0752debdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:41:40.389271 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.389223 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-tmp-dir\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:41:40.389271 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.389242 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-model-cache\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:41:40.389271 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.389260 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-home" (OuterVolumeSpecName: "home") pod "23238b6a-d33e-46b1-89b2-2fb0752debdb" (UID: "23238b6a-d33e-46b1-89b2-2fb0752debdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:41:40.391416 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.391391 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-dshm" (OuterVolumeSpecName: "dshm") pod "23238b6a-d33e-46b1-89b2-2fb0752debdb" (UID: "23238b6a-d33e-46b1-89b2-2fb0752debdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:41:40.391810 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.391789 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23238b6a-d33e-46b1-89b2-2fb0752debdb-kube-api-access-dwbxm" (OuterVolumeSpecName: "kube-api-access-dwbxm") pod "23238b6a-d33e-46b1-89b2-2fb0752debdb" (UID: "23238b6a-d33e-46b1-89b2-2fb0752debdb"). InnerVolumeSpecName "kube-api-access-dwbxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:41:40.391905 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.391869 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23238b6a-d33e-46b1-89b2-2fb0752debdb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23238b6a-d33e-46b1-89b2-2fb0752debdb" (UID: "23238b6a-d33e-46b1-89b2-2fb0752debdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:41:40.490661 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.490623 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-dshm\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:41:40.490661 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.490650 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23238b6a-d33e-46b1-89b2-2fb0752debdb-tls-certs\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:41:40.490661 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.490660 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbxm\" (UniqueName: \"kubernetes.io/projected/23238b6a-d33e-46b1-89b2-2fb0752debdb-kube-api-access-dwbxm\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:41:40.490661 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:40.490667 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23238b6a-d33e-46b1-89b2-2fb0752debdb-home\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:41:41.004333 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.004299 2569 generic.go:356] "Generic (PLEG): container finished" podID="23238b6a-d33e-46b1-89b2-2fb0752debdb" containerID="eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8" exitCode=0 Jul 08 19:41:41.004333 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.004337 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" event={"ID":"23238b6a-d33e-46b1-89b2-2fb0752debdb","Type":"ContainerDied","Data":"eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8"} Jul 08 19:41:41.004584 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.004343 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" Jul 08 19:41:41.004584 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.004364 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn" event={"ID":"23238b6a-d33e-46b1-89b2-2fb0752debdb","Type":"ContainerDied","Data":"ed5dd276321e8d7bed3f6a889c1de784d5cd34e3ced226348395c657ba5fbf49"} Jul 08 19:41:41.004584 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.004380 2569 scope.go:117] "RemoveContainer" containerID="eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8" Jul 08 19:41:41.013450 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.013430 2569 scope.go:117] "RemoveContainer" containerID="eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8" Jul 08 19:41:41.013734 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:41:41.013713 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8\": container with ID starting with eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8 not found: ID does not exist" containerID="eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8" Jul 08 19:41:41.013796 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.013743 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8"} err="failed to get container status \"eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8\": rpc error: code = NotFound desc = could not find container \"eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8\": container with ID starting with eaa937e519c4672a4c654164fc40845b825d8a5b08f96be1e6933f810a739ec8 not found: ID does not exist" Jul 08 19:41:41.025526 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.025501 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn"] Jul 08 19:41:41.029516 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.029494 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/auth-test-model-d86987-kserve-76769d6c95-ddnmn"] Jul 08 19:41:41.100352 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:41.100312 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23238b6a-d33e-46b1-89b2-2fb0752debdb" path="/var/lib/kubelet/pods/23238b6a-d33e-46b1-89b2-2fb0752debdb/volumes" Jul 08 19:41:43.922983 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:43.922929 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:43.923375 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:43.923004 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:43.935394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:43.935372 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:44.031477 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:44.031441 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:41:44.612405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:41:44.612371 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:42:08.707054 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.707019 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s"] Jul 08 19:42:08.707551 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.707414 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23238b6a-d33e-46b1-89b2-2fb0752debdb" containerName="main" Jul 08 19:42:08.707551 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.707426 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23238b6a-d33e-46b1-89b2-2fb0752debdb" containerName="main" Jul 08 19:42:08.707551 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.707484 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="23238b6a-d33e-46b1-89b2-2fb0752debdb" containerName="main" Jul 08 19:42:08.710571 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.710530 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.713986 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.713963 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"default-dockercfg-kcgp9\"" Jul 08 19:42:08.714113 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.714010 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"openshift-service-ca.crt\"" Jul 08 19:42:08.714113 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.714034 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"sub-test-model-4a9cb4-kserve-self-signed-certs\"" Jul 08 19:42:08.714113 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.714019 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"kube-root-ca.crt\"" Jul 08 19:42:08.719058 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.719037 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s"] Jul 08 19:42:08.847394 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.847365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-dshm\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.847585 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.847412 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-model-cache\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.847585 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.847481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d56z\" (UniqueName: \"kubernetes.io/projected/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-kube-api-access-9d56z\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.847585 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.847503 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tmp-dir\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.847702 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.847597 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tls-certs\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.847702 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.847634 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-home\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948192 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d56z\" (UniqueName: \"kubernetes.io/projected/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-kube-api-access-9d56z\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948345 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tmp-dir\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948345 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tls-certs\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948345 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-home\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948447 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948425 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-dshm\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948522 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948503 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-model-cache\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948673 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948655 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tmp-dir\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948746 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948697 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-home\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.948873 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.948853 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-model-cache\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.950745 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.950716 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-dshm\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.950974 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.950958 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tls-certs\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:08.956609 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:08.956590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d56z\" (UniqueName: \"kubernetes.io/projected/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-kube-api-access-9d56z\") pod \"sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:09.021026 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:09.020945 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:09.161524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:09.161484 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s"] Jul 08 19:42:09.169721 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:42:09.169689 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d20b3f1_3ce4_4e61_9718_423f91c3f2a8.slice/crio-d7576e531b0f5c64cbbbd80c397d190618ef8354d76c4f87c4c5f015749054e1 WatchSource:0}: Error finding container d7576e531b0f5c64cbbbd80c397d190618ef8354d76c4f87c4c5f015749054e1: Status 404 returned error can't find the container with id d7576e531b0f5c64cbbbd80c397d190618ef8354d76c4f87c4c5f015749054e1 Jul 08 19:42:10.144174 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:10.144132 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" event={"ID":"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8","Type":"ContainerStarted","Data":"29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f"} Jul 08 19:42:10.144174 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:10.144173 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" event={"ID":"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8","Type":"ContainerStarted","Data":"d7576e531b0f5c64cbbbd80c397d190618ef8354d76c4f87c4c5f015749054e1"} Jul 08 19:42:10.165837 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:10.165772 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" podStartSLOduration=2.165756852 podStartE2EDuration="2.165756852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:10.163666007 +0000 UTC m=+2275.602968638" watchObservedRunningTime="2026-07-08 19:42:10.165756852 +0000 UTC m=+2275.605059506" Jul 08 19:42:13.302975 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:13.302945 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:42:19.021182 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:19.021135 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:19.021604 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:19.021203 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:19.033880 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:19.033852 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:19.201242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:19.201210 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:42:30.303265 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:30.303227 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:42:49.356277 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:49.356246 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt"] Jul 08 19:42:50.405430 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:50.405391 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s"] Jul 08 19:42:51.219909 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:42:51.219876 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:43:04.376274 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.376214 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" podUID="883ba18f-4bba-436e-979f-16e634f99fa4" containerName="main" containerID="cri-o://09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68" gracePeriod=45 Jul 08 19:43:04.630111 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.630035 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:43:04.684068 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684040 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-dshm\") pod \"883ba18f-4bba-436e-979f-16e634f99fa4\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " Jul 08 19:43:04.684227 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684096 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tk9h\" (UniqueName: \"kubernetes.io/projected/883ba18f-4bba-436e-979f-16e634f99fa4-kube-api-access-8tk9h\") pod \"883ba18f-4bba-436e-979f-16e634f99fa4\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " Jul 08 19:43:04.684227 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684114 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883ba18f-4bba-436e-979f-16e634f99fa4-tls-certs\") pod \"883ba18f-4bba-436e-979f-16e634f99fa4\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " Jul 08 19:43:04.684227 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684187 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-model-cache\") pod \"883ba18f-4bba-436e-979f-16e634f99fa4\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " Jul 08 19:43:04.684227 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684217 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-tmp-dir\") pod \"883ba18f-4bba-436e-979f-16e634f99fa4\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " Jul 08 19:43:04.684457 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684236 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-home\") pod \"883ba18f-4bba-436e-979f-16e634f99fa4\" (UID: \"883ba18f-4bba-436e-979f-16e634f99fa4\") " Jul 08 19:43:04.684517 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684438 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "883ba18f-4bba-436e-979f-16e634f99fa4" (UID: "883ba18f-4bba-436e-979f-16e634f99fa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:04.684517 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684470 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-model-cache" (OuterVolumeSpecName: "model-cache") pod "883ba18f-4bba-436e-979f-16e634f99fa4" (UID: "883ba18f-4bba-436e-979f-16e634f99fa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:04.684640 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684528 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-home" (OuterVolumeSpecName: "home") pod "883ba18f-4bba-436e-979f-16e634f99fa4" (UID: "883ba18f-4bba-436e-979f-16e634f99fa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:04.684681 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684646 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-tmp-dir\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:04.684681 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.684659 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-model-cache\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:04.686244 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.686221 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-dshm" (OuterVolumeSpecName: "dshm") pod "883ba18f-4bba-436e-979f-16e634f99fa4" (UID: "883ba18f-4bba-436e-979f-16e634f99fa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:04.686743 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.686727 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883ba18f-4bba-436e-979f-16e634f99fa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "883ba18f-4bba-436e-979f-16e634f99fa4" (UID: "883ba18f-4bba-436e-979f-16e634f99fa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:43:04.686831 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.686741 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883ba18f-4bba-436e-979f-16e634f99fa4-kube-api-access-8tk9h" (OuterVolumeSpecName: "kube-api-access-8tk9h") pod "883ba18f-4bba-436e-979f-16e634f99fa4" (UID: "883ba18f-4bba-436e-979f-16e634f99fa4"). InnerVolumeSpecName "kube-api-access-8tk9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:43:04.785815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.785782 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-dshm\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:04.785815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.785807 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tk9h\" (UniqueName: \"kubernetes.io/projected/883ba18f-4bba-436e-979f-16e634f99fa4-kube-api-access-8tk9h\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:04.785815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.785817 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883ba18f-4bba-436e-979f-16e634f99fa4-tls-certs\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:04.785815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:04.785827 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883ba18f-4bba-436e-979f-16e634f99fa4-home\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:05.416249 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.416210 2569 generic.go:356] "Generic (PLEG): container finished" podID="883ba18f-4bba-436e-979f-16e634f99fa4" containerID="09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68" exitCode=0 Jul 08 19:43:05.416685 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.416258 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" Jul 08 19:43:05.416685 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.416295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" event={"ID":"883ba18f-4bba-436e-979f-16e634f99fa4","Type":"ContainerDied","Data":"09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68"} Jul 08 19:43:05.416685 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.416341 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt" event={"ID":"883ba18f-4bba-436e-979f-16e634f99fa4","Type":"ContainerDied","Data":"1bd47f667931ecfe7ab36fe328835f9a5f4e63a577c6997da14e6b1826e01f1c"} Jul 08 19:43:05.416685 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.416359 2569 scope.go:117] "RemoveContainer" containerID="09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68" Jul 08 19:43:05.425878 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.425845 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" podUID="5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" containerName="main" containerID="cri-o://29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f" gracePeriod=45 Jul 08 19:43:05.431139 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.431117 2569 scope.go:117] "RemoveContainer" containerID="09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68" Jul 08 19:43:05.431433 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:43:05.431403 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68\": container with ID starting with 09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68 not found: ID does not exist" containerID="09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68" Jul 08 19:43:05.431518 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.431440 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68"} err="failed to get container status \"09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68\": rpc error: code = NotFound desc = could not find container \"09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68\": container with ID starting with 09832923627aae17b5327a5286046499e6fcbc9ab8db59cde1e0a7af73492a68 not found: ID does not exist" Jul 08 19:43:05.442239 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.442218 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt"] Jul 08 19:43:05.445733 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.445714 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d86987/sub-test-model-d86987-kserve-5b87c87c5-qrtzt"] Jul 08 19:43:05.679962 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.679907 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:43:05.694531 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694506 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-dshm\") pod \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " Jul 08 19:43:05.694668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694572 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-model-cache\") pod \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " Jul 08 19:43:05.694668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694600 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-home\") pod \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " Jul 08 19:43:05.694668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694636 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d56z\" (UniqueName: \"kubernetes.io/projected/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-kube-api-access-9d56z\") pod \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " Jul 08 19:43:05.694668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694654 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tmp-dir\") pod \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " Jul 08 19:43:05.694845 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694681 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tls-certs\") pod \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\" (UID: \"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8\") " Jul 08 19:43:05.694926 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694882 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" (UID: "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:05.694990 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694919 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-home" (OuterVolumeSpecName: "home") pod "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" (UID: "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:05.694990 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.694934 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" (UID: "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:05.695116 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.695036 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tmp-dir\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:05.695116 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.695062 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-model-cache\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:05.695116 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.695076 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-home\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:05.696825 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.696803 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-dshm" (OuterVolumeSpecName: "dshm") pod "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" (UID: "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:05.696948 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.696847 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" (UID: "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:43:05.696948 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.696936 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-kube-api-access-9d56z" (OuterVolumeSpecName: "kube-api-access-9d56z") pod "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" (UID: "5d20b3f1-3ce4-4e61-9718-423f91c3f2a8"). InnerVolumeSpecName "kube-api-access-9d56z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:43:05.795815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.795779 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d56z\" (UniqueName: \"kubernetes.io/projected/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-kube-api-access-9d56z\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:05.795815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.795811 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-tls-certs\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:05.795815 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:05.795822 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8-dshm\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:43:06.423491 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.423458 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" containerID="29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f" exitCode=0 Jul 08 19:43:06.423978 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.423506 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" Jul 08 19:43:06.423978 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.423553 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" event={"ID":"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8","Type":"ContainerDied","Data":"29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f"} Jul 08 19:43:06.423978 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.423588 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s" event={"ID":"5d20b3f1-3ce4-4e61-9718-423f91c3f2a8","Type":"ContainerDied","Data":"d7576e531b0f5c64cbbbd80c397d190618ef8354d76c4f87c4c5f015749054e1"} Jul 08 19:43:06.423978 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.423606 2569 scope.go:117] "RemoveContainer" containerID="29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f" Jul 08 19:43:06.433310 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.433297 2569 scope.go:117] "RemoveContainer" containerID="29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f" Jul 08 19:43:06.433601 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:43:06.433578 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f\": container with ID starting with 29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f not found: ID does not exist" containerID="29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f" Jul 08 19:43:06.433689 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.433611 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f"} err="failed to get container status \"29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f\": rpc error: code = NotFound desc = could not find container \"29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f\": container with ID starting with 29827b7fe97682f2766e515999c62426553f940b7c7250dffb8e006dd8cff30f not found: ID does not exist" Jul 08 19:43:06.444300 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.444277 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s"] Jul 08 19:43:06.446035 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:06.446013 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/sub-test-model-4a9cb4-kserve-bfdb7d8cc-jw48s"] Jul 08 19:43:07.100579 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:07.100527 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" path="/var/lib/kubelet/pods/5d20b3f1-3ce4-4e61-9718-423f91c3f2a8/volumes" Jul 08 19:43:07.100961 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:07.100947 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883ba18f-4bba-436e-979f-16e634f99fa4" path="/var/lib/kubelet/pods/883ba18f-4bba-436e-979f-16e634f99fa4/volumes" Jul 08 19:43:13.704009 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:13.703973 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:43:47.117853 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.117818 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v"] Jul 08 19:43:47.118309 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.118190 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" containerName="main" Jul 08 19:43:47.118309 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.118200 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" containerName="main" Jul 08 19:43:47.118309 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.118216 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883ba18f-4bba-436e-979f-16e634f99fa4" containerName="main" Jul 08 19:43:47.118309 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.118223 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="883ba18f-4bba-436e-979f-16e634f99fa4" containerName="main" Jul 08 19:43:47.118309 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.118287 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d20b3f1-3ce4-4e61-9718-423f91c3f2a8" containerName="main" Jul 08 19:43:47.118309 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.118296 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="883ba18f-4bba-436e-979f-16e634f99fa4" containerName="main" Jul 08 19:43:47.120920 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.120904 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.123404 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.123379 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"openshift-service-ca.crt\"" Jul 08 19:43:47.124456 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.124409 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"kube-root-ca.crt\"" Jul 08 19:43:47.124456 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.124409 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"default-dockercfg-kcgp9\"" Jul 08 19:43:47.124456 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.124448 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a9cb4\"/\"rate-test-model-4a9cb4-kserve-self-signed-certs\"" Jul 08 19:43:47.130901 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.130879 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v"] Jul 08 19:43:47.189122 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.189088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-home\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.189273 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.189206 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-model-cache\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.189273 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.189237 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8mqx\" (UniqueName: \"kubernetes.io/projected/8eefd09c-b502-4d86-887c-d658ba2f1074-kube-api-access-f8mqx\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.189362 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.189290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-dshm\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.189362 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.189323 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-tmp-dir\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.189362 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.189343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eefd09c-b502-4d86-887c-d658ba2f1074-tls-certs\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.290827 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.290795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-model-cache\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.290838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8mqx\" (UniqueName: \"kubernetes.io/projected/8eefd09c-b502-4d86-887c-d658ba2f1074-kube-api-access-f8mqx\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.290880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-dshm\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.290910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-tmp-dir\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.290931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eefd09c-b502-4d86-887c-d658ba2f1074-tls-certs\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.290965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-home\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291297 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.291263 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-model-cache\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291366 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.291349 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-tmp-dir\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.291426 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.291410 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-home\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.293182 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.293161 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-dshm\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.293429 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.293411 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eefd09c-b502-4d86-887c-d658ba2f1074-tls-certs\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.304555 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.304503 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8mqx\" (UniqueName: \"kubernetes.io/projected/8eefd09c-b502-4d86-887c-d658ba2f1074-kube-api-access-f8mqx\") pod \"rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.431232 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.431191 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:47.568697 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.568672 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v"] Jul 08 19:43:47.571052 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:43:47.571028 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8eefd09c_b502_4d86_887c_d658ba2f1074.slice/crio-b6be5b4f7239000dd3a0230bfd47dbea2b16566b8c5d654b2f4ab74545b4f304 WatchSource:0}: Error finding container b6be5b4f7239000dd3a0230bfd47dbea2b16566b8c5d654b2f4ab74545b4f304: Status 404 returned error can't find the container with id b6be5b4f7239000dd3a0230bfd47dbea2b16566b8c5d654b2f4ab74545b4f304 Jul 08 19:43:47.625405 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:47.625373 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" event={"ID":"8eefd09c-b502-4d86-887c-d658ba2f1074","Type":"ContainerStarted","Data":"b6be5b4f7239000dd3a0230bfd47dbea2b16566b8c5d654b2f4ab74545b4f304"} Jul 08 19:43:48.632334 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:48.632297 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" event={"ID":"8eefd09c-b502-4d86-887c-d658ba2f1074","Type":"ContainerStarted","Data":"24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf"} Jul 08 19:43:48.651092 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:48.651043 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" podStartSLOduration=1.65102624 podStartE2EDuration="1.65102624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:43:48.649286844 +0000 UTC m=+2374.088589488" watchObservedRunningTime="2026-07-08 19:43:48.65102624 +0000 UTC m=+2374.090328880" Jul 08 19:43:51.910968 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:51.910930 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:43:57.431354 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:57.431318 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:57.431782 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:57.431371 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:57.443864 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:57.443838 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:43:57.687793 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:43:57.687707 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:44:06.815694 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:06.815657 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:44:15.153226 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:15.153194 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:44:15.155608 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:15.155588 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbj8q_5157aa65-f232-4a4b-8f18-42cfb4ba47ff/ovn-acl-logging/0.log" Jul 08 19:44:27.211280 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:27.211242 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:44:32.109511 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:32.109479 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v"] Jul 08 19:44:33.049668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.049634 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387355824"] Jul 08 19:44:33.053189 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.053172 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:33.055652 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.055632 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 19:44:33.056389 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.056370 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 19:44:33.056389 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.056388 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w7929\"" Jul 08 19:44:33.059713 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.059692 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387355824"] Jul 08 19:44:33.130961 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.130926 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzzf\" (UniqueName: \"kubernetes.io/projected/67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2-kube-api-access-lxzzf\") pod \"test-curl-7660-140441387355824\" (UID: \"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2\") " pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:33.231640 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.231605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzzf\" (UniqueName: \"kubernetes.io/projected/67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2-kube-api-access-lxzzf\") pod \"test-curl-7660-140441387355824\" (UID: \"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2\") " pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:33.240031 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.240002 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzzf\" (UniqueName: \"kubernetes.io/projected/67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2-kube-api-access-lxzzf\") pod \"test-curl-7660-140441387355824\" (UID: \"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2\") " pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:33.363327 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.363252 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:33.493270 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.493245 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387355824"] Jul 08 19:44:33.496207 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:44:33.496174 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67ae2935_0a54_4bf3_9c53_27f1b1e1c7b2.slice/crio-0c07277f8868037ea0ade7aafdeaf2e430aad5e148a26615c0c2174a9286246c WatchSource:0}: Error finding container 0c07277f8868037ea0ade7aafdeaf2e430aad5e148a26615c0c2174a9286246c: Status 404 returned error can't find the container with id 0c07277f8868037ea0ade7aafdeaf2e430aad5e148a26615c0c2174a9286246c Jul 08 19:44:33.497506 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.497490 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:44:33.857605 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:33.857569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" event={"ID":"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2","Type":"ContainerStarted","Data":"0c07277f8868037ea0ade7aafdeaf2e430aad5e148a26615c0c2174a9286246c"} Jul 08 19:44:34.863650 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:34.863615 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" event={"ID":"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2","Type":"ContainerStarted","Data":"030795d8c4bb656744ef18a4b44de44f66eb155bcd69efd29abde111ad5aa118"} Jul 08 19:44:34.879676 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:34.879631 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" podStartSLOduration=1.122860389 podStartE2EDuration="1.879614833s" totalImagesPullingTime="756.754444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:44:34.878437031 +0000 UTC m=+2420.317739662" watchObservedRunningTime="2026-07-08 19:44:34.879614833 +0000 UTC m=+2420.318917464" Jul 08 19:44:44.808065 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:44.808024 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:44:44.913508 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:44.913476 2569 generic.go:356] "Generic (PLEG): container finished" podID="67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2" containerID="030795d8c4bb656744ef18a4b44de44f66eb155bcd69efd29abde111ad5aa118" exitCode=28 Jul 08 19:44:44.913700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:44.913573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" event={"ID":"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2","Type":"ContainerDied","Data":"030795d8c4bb656744ef18a4b44de44f66eb155bcd69efd29abde111ad5aa118"} Jul 08 19:44:46.058259 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.058231 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:46.089160 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.089127 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387355824"] Jul 08 19:44:46.091219 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.091195 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387355824"] Jul 08 19:44:46.153396 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.153369 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxzzf\" (UniqueName: \"kubernetes.io/projected/67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2-kube-api-access-lxzzf\") pod \"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2\" (UID: \"67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2\") " Jul 08 19:44:46.155670 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.155648 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2-kube-api-access-lxzzf" (OuterVolumeSpecName: "kube-api-access-lxzzf") pod "67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2" (UID: "67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2"). InnerVolumeSpecName "kube-api-access-lxzzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:44:46.255164 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.255076 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxzzf\" (UniqueName: \"kubernetes.io/projected/67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2-kube-api-access-lxzzf\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:46.258322 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.258296 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387353136"] Jul 08 19:44:46.258697 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.258684 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2" containerName="test-curl-7660-140441387355824" Jul 08 19:44:46.258748 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.258699 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2" containerName="test-curl-7660-140441387355824" Jul 08 19:44:46.258788 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.258763 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2" containerName="test-curl-7660-140441387355824" Jul 08 19:44:46.262357 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.262341 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:44:46.265615 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.265595 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387353136"] Jul 08 19:44:46.355932 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.355900 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98czx\" (UniqueName: \"kubernetes.io/projected/46151709-003a-46b0-bba5-59a295bc0317-kube-api-access-98czx\") pod \"test-curl-7660-140441387353136\" (UID: \"46151709-003a-46b0-bba5-59a295bc0317\") " pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:44:46.457250 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.457207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98czx\" (UniqueName: \"kubernetes.io/projected/46151709-003a-46b0-bba5-59a295bc0317-kube-api-access-98czx\") pod \"test-curl-7660-140441387353136\" (UID: \"46151709-003a-46b0-bba5-59a295bc0317\") " pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:44:46.464809 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.464784 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98czx\" (UniqueName: \"kubernetes.io/projected/46151709-003a-46b0-bba5-59a295bc0317-kube-api-access-98czx\") pod \"test-curl-7660-140441387353136\" (UID: \"46151709-003a-46b0-bba5-59a295bc0317\") " pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:44:46.573036 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.572949 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:44:46.704164 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.704139 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387353136"] Jul 08 19:44:46.706094 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:44:46.706072 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46151709_003a_46b0_bba5_59a295bc0317.slice/crio-6bd21984a6f746ba1a48cb7361daffac8c518fd324f2d6178eb0999978b9ae90 WatchSource:0}: Error finding container 6bd21984a6f746ba1a48cb7361daffac8c518fd324f2d6178eb0999978b9ae90: Status 404 returned error can't find the container with id 6bd21984a6f746ba1a48cb7361daffac8c518fd324f2d6178eb0999978b9ae90 Jul 08 19:44:46.924384 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.924350 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" event={"ID":"46151709-003a-46b0-bba5-59a295bc0317","Type":"ContainerStarted","Data":"6bd21984a6f746ba1a48cb7361daffac8c518fd324f2d6178eb0999978b9ae90"} Jul 08 19:44:46.925513 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.925494 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c07277f8868037ea0ade7aafdeaf2e430aad5e148a26615c0c2174a9286246c" Jul 08 19:44:46.925631 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:46.925527 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:46.935342 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:44:46.935313 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-140441387355824\" is forbidden: User \"system:node:ip-10-0-140-33.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-33.ec2.internal' and this object" podUID="67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2" pod="odh-ai-gateway-infra/test-curl-7660-140441387355824" Jul 08 19:44:47.099866 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.099832 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2" path="/var/lib/kubelet/pods/67ae2935-0a54-4bf3-9c53-27f1b1e1c7b2/volumes" Jul 08 19:44:47.130380 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.130345 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" podUID="8eefd09c-b502-4d86-887c-d658ba2f1074" containerName="main" containerID="cri-o://24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf" gracePeriod=45 Jul 08 19:44:47.383700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.383670 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:44:47.466772 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.466742 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-home\") pod \"8eefd09c-b502-4d86-887c-d658ba2f1074\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " Jul 08 19:44:47.466945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.466791 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-dshm\") pod \"8eefd09c-b502-4d86-887c-d658ba2f1074\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " Jul 08 19:44:47.466945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.466824 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8mqx\" (UniqueName: \"kubernetes.io/projected/8eefd09c-b502-4d86-887c-d658ba2f1074-kube-api-access-f8mqx\") pod \"8eefd09c-b502-4d86-887c-d658ba2f1074\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " Jul 08 19:44:47.466945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.466855 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-model-cache\") pod \"8eefd09c-b502-4d86-887c-d658ba2f1074\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " Jul 08 19:44:47.466945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.466887 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eefd09c-b502-4d86-887c-d658ba2f1074-tls-certs\") pod \"8eefd09c-b502-4d86-887c-d658ba2f1074\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " Jul 08 19:44:47.466945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.466921 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-tmp-dir\") pod \"8eefd09c-b502-4d86-887c-d658ba2f1074\" (UID: \"8eefd09c-b502-4d86-887c-d658ba2f1074\") " Jul 08 19:44:47.467227 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.467069 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-home" (OuterVolumeSpecName: "home") pod "8eefd09c-b502-4d86-887c-d658ba2f1074" (UID: "8eefd09c-b502-4d86-887c-d658ba2f1074"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:44:47.467227 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.467162 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-model-cache" (OuterVolumeSpecName: "model-cache") pod "8eefd09c-b502-4d86-887c-d658ba2f1074" (UID: "8eefd09c-b502-4d86-887c-d658ba2f1074"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:44:47.467322 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.467287 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8eefd09c-b502-4d86-887c-d658ba2f1074" (UID: "8eefd09c-b502-4d86-887c-d658ba2f1074"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:44:47.467367 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.467317 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-home\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:47.467367 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.467338 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-model-cache\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:47.469087 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.469058 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eefd09c-b502-4d86-887c-d658ba2f1074-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8eefd09c-b502-4d86-887c-d658ba2f1074" (UID: "8eefd09c-b502-4d86-887c-d658ba2f1074"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:44:47.469189 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.469096 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eefd09c-b502-4d86-887c-d658ba2f1074-kube-api-access-f8mqx" (OuterVolumeSpecName: "kube-api-access-f8mqx") pod "8eefd09c-b502-4d86-887c-d658ba2f1074" (UID: "8eefd09c-b502-4d86-887c-d658ba2f1074"). InnerVolumeSpecName "kube-api-access-f8mqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:44:47.469242 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.469229 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-dshm" (OuterVolumeSpecName: "dshm") pod "8eefd09c-b502-4d86-887c-d658ba2f1074" (UID: "8eefd09c-b502-4d86-887c-d658ba2f1074"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:44:47.568485 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.568401 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eefd09c-b502-4d86-887c-d658ba2f1074-tls-certs\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:47.568485 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.568428 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-tmp-dir\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:47.568485 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.568437 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eefd09c-b502-4d86-887c-d658ba2f1074-dshm\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:47.568485 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.568446 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8mqx\" (UniqueName: \"kubernetes.io/projected/8eefd09c-b502-4d86-887c-d658ba2f1074-kube-api-access-f8mqx\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:47.931873 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.931834 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" event={"ID":"46151709-003a-46b0-bba5-59a295bc0317","Type":"ContainerStarted","Data":"6a1648a1d11979dffaa44edd1c6d288ee7b99b75756c023596570055cb155cc1"} Jul 08 19:44:47.933106 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.933085 2569 generic.go:356] "Generic (PLEG): container finished" podID="8eefd09c-b502-4d86-887c-d658ba2f1074" containerID="24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf" exitCode=0 Jul 08 19:44:47.933177 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.933136 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" Jul 08 19:44:47.933177 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.933162 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" event={"ID":"8eefd09c-b502-4d86-887c-d658ba2f1074","Type":"ContainerDied","Data":"24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf"} Jul 08 19:44:47.933260 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.933194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v" event={"ID":"8eefd09c-b502-4d86-887c-d658ba2f1074","Type":"ContainerDied","Data":"b6be5b4f7239000dd3a0230bfd47dbea2b16566b8c5d654b2f4ab74545b4f304"} Jul 08 19:44:47.933260 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.933211 2569 scope.go:117] "RemoveContainer" containerID="24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf" Jul 08 19:44:47.942660 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.942641 2569 scope.go:117] "RemoveContainer" containerID="24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf" Jul 08 19:44:47.942878 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:44:47.942861 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf\": container with ID starting with 24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf not found: ID does not exist" containerID="24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf" Jul 08 19:44:47.942925 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.942885 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf"} err="failed to get container status \"24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf\": rpc error: code = NotFound desc = could not find container \"24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf\": container with ID starting with 24ac2eccedeff4ff22cbdf97f3f496fcee1aa49146531fc58bec77f95a6ecdcf not found: ID does not exist" Jul 08 19:44:47.947503 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.947465 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" podStartSLOduration=1.654811877 podStartE2EDuration="1.947452035s" totalImagesPullingTime="292.640158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:44:47.946336674 +0000 UTC m=+2433.385639307" watchObservedRunningTime="2026-07-08 19:44:47.947452035 +0000 UTC m=+2433.386754665" Jul 08 19:44:47.960170 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.960141 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v"] Jul 08 19:44:47.964352 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:47.964327 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4a9cb4/rate-test-model-4a9cb4-kserve-65b744c7c9-kj85v"] Jul 08 19:44:49.099601 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:49.099565 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eefd09c-b502-4d86-887c-d658ba2f1074" path="/var/lib/kubelet/pods/8eefd09c-b502-4d86-887c-d658ba2f1074/volumes" Jul 08 19:44:57.983191 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:57.983157 2569 generic.go:356] "Generic (PLEG): container finished" podID="46151709-003a-46b0-bba5-59a295bc0317" containerID="6a1648a1d11979dffaa44edd1c6d288ee7b99b75756c023596570055cb155cc1" exitCode=28 Jul 08 19:44:57.983191 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:57.983192 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" event={"ID":"46151709-003a-46b0-bba5-59a295bc0317","Type":"ContainerDied","Data":"6a1648a1d11979dffaa44edd1c6d288ee7b99b75756c023596570055cb155cc1"} Jul 08 19:44:59.132070 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.132043 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:44:59.162267 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.162232 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387353136"] Jul 08 19:44:59.166005 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.165979 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441387353136"] Jul 08 19:44:59.176757 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.176726 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98czx\" (UniqueName: \"kubernetes.io/projected/46151709-003a-46b0-bba5-59a295bc0317-kube-api-access-98czx\") pod \"46151709-003a-46b0-bba5-59a295bc0317\" (UID: \"46151709-003a-46b0-bba5-59a295bc0317\") " Jul 08 19:44:59.178663 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.178642 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46151709-003a-46b0-bba5-59a295bc0317-kube-api-access-98czx" (OuterVolumeSpecName: "kube-api-access-98czx") pod "46151709-003a-46b0-bba5-59a295bc0317" (UID: "46151709-003a-46b0-bba5-59a295bc0317"). InnerVolumeSpecName "kube-api-access-98czx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:44:59.278433 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.278353 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98czx\" (UniqueName: \"kubernetes.io/projected/46151709-003a-46b0-bba5-59a295bc0317-kube-api-access-98czx\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:44:59.328532 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.328502 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441388650832"] Jul 08 19:44:59.328895 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.328882 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eefd09c-b502-4d86-887c-d658ba2f1074" containerName="main" Jul 08 19:44:59.328945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.328897 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eefd09c-b502-4d86-887c-d658ba2f1074" containerName="main" Jul 08 19:44:59.328945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.328919 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46151709-003a-46b0-bba5-59a295bc0317" containerName="test-curl-7660-140441387353136" Jul 08 19:44:59.328945 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.328926 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="46151709-003a-46b0-bba5-59a295bc0317" containerName="test-curl-7660-140441387353136" Jul 08 19:44:59.329036 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.328986 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eefd09c-b502-4d86-887c-d658ba2f1074" containerName="main" Jul 08 19:44:59.329036 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.328993 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="46151709-003a-46b0-bba5-59a295bc0317" containerName="test-curl-7660-140441387353136" Jul 08 19:44:59.332033 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.332017 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:44:59.340265 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.340242 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441388650832"] Jul 08 19:44:59.379133 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.379107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsqn\" (UniqueName: \"kubernetes.io/projected/3ca98722-cbf7-4fb7-b503-069aca87f924-kube-api-access-jgsqn\") pod \"test-curl-7660-140441388650832\" (UID: \"3ca98722-cbf7-4fb7-b503-069aca87f924\") " pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:44:59.479549 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.479511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsqn\" (UniqueName: \"kubernetes.io/projected/3ca98722-cbf7-4fb7-b503-069aca87f924-kube-api-access-jgsqn\") pod \"test-curl-7660-140441388650832\" (UID: \"3ca98722-cbf7-4fb7-b503-069aca87f924\") " pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:44:59.487589 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.487560 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsqn\" (UniqueName: \"kubernetes.io/projected/3ca98722-cbf7-4fb7-b503-069aca87f924-kube-api-access-jgsqn\") pod \"test-curl-7660-140441388650832\" (UID: \"3ca98722-cbf7-4fb7-b503-069aca87f924\") " pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:44:59.641610 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.641501 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:44:59.773711 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.773682 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441388650832"] Jul 08 19:44:59.777029 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:44:59.777000 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca98722_cbf7_4fb7_b503_069aca87f924.slice/crio-06300ebc1d4a652728170048c795a32df7a67bcaabbc2df8c48b31cabc4ed2de WatchSource:0}: Error finding container 06300ebc1d4a652728170048c795a32df7a67bcaabbc2df8c48b31cabc4ed2de: Status 404 returned error can't find the container with id 06300ebc1d4a652728170048c795a32df7a67bcaabbc2df8c48b31cabc4ed2de Jul 08 19:44:59.994471 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.994441 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:44:59.994668 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.994441 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd21984a6f746ba1a48cb7361daffac8c518fd324f2d6178eb0999978b9ae90" Jul 08 19:44:59.995660 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:44:59.995632 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" event={"ID":"3ca98722-cbf7-4fb7-b503-069aca87f924","Type":"ContainerStarted","Data":"06300ebc1d4a652728170048c795a32df7a67bcaabbc2df8c48b31cabc4ed2de"} Jul 08 19:45:00.008241 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:45:00.008213 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-140441387353136\" is forbidden: User \"system:node:ip-10-0-140-33.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-33.ec2.internal' and this object" podUID="46151709-003a-46b0-bba5-59a295bc0317" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:45:01.002478 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:01.002439 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" event={"ID":"3ca98722-cbf7-4fb7-b503-069aca87f924","Type":"ContainerStarted","Data":"841e7d3a29a15b0ffaae358fe7a364e9f3427c026c1e82c1d43e7b6ba4180194"} Jul 08 19:45:01.004631 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:45:01.004605 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-140441387353136\" is forbidden: User \"system:node:ip-10-0-140-33.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-33.ec2.internal' and this object" podUID="46151709-003a-46b0-bba5-59a295bc0317" pod="odh-ai-gateway-infra/test-curl-7660-140441387353136" Jul 08 19:45:01.018996 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:01.018944 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" podStartSLOduration=1.6869165640000001 podStartE2EDuration="2.018930114s" totalImagesPullingTime="332.01355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:45:01.018300647 +0000 UTC m=+2446.457603281" watchObservedRunningTime="2026-07-08 19:45:01.018930114 +0000 UTC m=+2446.458232745" Jul 08 19:45:01.100102 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:01.100072 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46151709-003a-46b0-bba5-59a295bc0317" path="/var/lib/kubelet/pods/46151709-003a-46b0-bba5-59a295bc0317/volumes" Jul 08 19:45:11.054360 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:11.054323 2569 generic.go:356] "Generic (PLEG): container finished" podID="3ca98722-cbf7-4fb7-b503-069aca87f924" containerID="841e7d3a29a15b0ffaae358fe7a364e9f3427c026c1e82c1d43e7b6ba4180194" exitCode=28 Jul 08 19:45:11.054793 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:11.054393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" event={"ID":"3ca98722-cbf7-4fb7-b503-069aca87f924","Type":"ContainerDied","Data":"841e7d3a29a15b0ffaae358fe7a364e9f3427c026c1e82c1d43e7b6ba4180194"} Jul 08 19:45:12.196431 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:12.196410 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:45:12.227753 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:12.227715 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441388650832"] Jul 08 19:45:12.231176 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:12.231148 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7660-140441388650832"] Jul 08 19:45:12.307947 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:12.307914 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgsqn\" (UniqueName: \"kubernetes.io/projected/3ca98722-cbf7-4fb7-b503-069aca87f924-kube-api-access-jgsqn\") pod \"3ca98722-cbf7-4fb7-b503-069aca87f924\" (UID: \"3ca98722-cbf7-4fb7-b503-069aca87f924\") " Jul 08 19:45:12.310129 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:12.310070 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca98722-cbf7-4fb7-b503-069aca87f924-kube-api-access-jgsqn" (OuterVolumeSpecName: "kube-api-access-jgsqn") pod "3ca98722-cbf7-4fb7-b503-069aca87f924" (UID: "3ca98722-cbf7-4fb7-b503-069aca87f924"). InnerVolumeSpecName "kube-api-access-jgsqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:45:12.409472 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:12.409442 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsqn\" (UniqueName: \"kubernetes.io/projected/3ca98722-cbf7-4fb7-b503-069aca87f924-kube-api-access-jgsqn\") on node \"ip-10-0-140-33.ec2.internal\" DevicePath \"\"" Jul 08 19:45:13.065580 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:13.065560 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06300ebc1d4a652728170048c795a32df7a67bcaabbc2df8c48b31cabc4ed2de" Jul 08 19:45:13.065580 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:13.065562 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:45:13.075193 ip-10-0-140-33 kubenswrapper[2569]: E0708 19:45:13.075163 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-140441388650832\" is forbidden: User \"system:node:ip-10-0-140-33.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-140-33.ec2.internal' and this object" podUID="3ca98722-cbf7-4fb7-b503-069aca87f924" pod="odh-ai-gateway-infra/test-curl-7660-140441388650832" Jul 08 19:45:13.099154 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:13.099131 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca98722-cbf7-4fb7-b503-069aca87f924" path="/var/lib/kubelet/pods/3ca98722-cbf7-4fb7-b503-069aca87f924/volumes" Jul 08 19:45:17.106034 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:17.105996 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:45:30.007701 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:30.007661 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wn8f4"] Jul 08 19:45:32.308726 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:32.308700 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-84g49_799e5671-ff3e-407b-a3ce-c9268e717ae1/copy-manifests/0.log" Jul 08 19:45:32.315593 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:32.315568 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-84g49_799e5671-ff3e-407b-a3ce-c9268e717ae1/manager/0.log" Jul 08 19:45:33.862031 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:33.861996 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xj977_458ac938-570c-402b-b44c-fd57cb3ef0ae/manager/0.log" Jul 08 19:45:33.976907 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:33.976879 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9t9fq_ca9c2331-85e4-4f8b-919c-e677298dd07d/kuadrant-console-plugin/0.log" Jul 08 19:45:34.324223 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:34.324198 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wn8f4_600443f6-e855-48ab-b5a0-411d96f04088/limitador/0.log" Jul 08 19:45:35.131599 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:35.131567 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdb786f-hkp72_9f18b0e9-db82-489a-b3c7-723eaa8e5cb8/kube-auth-proxy/0.log" Jul 08 19:45:36.191037 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:36.191004 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5f4vn_7ff6b9f9-6174-4df9-a854-ebb0a4744d40/main/0.log" Jul 08 19:45:36.528949 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:36.528842 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-82s5j_2cf4ce8b-c15f-457b-8da8-653a7e8f3a5a/main/0.log" Jul 08 19:45:43.245077 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:43.245044 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wc6c2_b1ce6677-e64a-4c21-9faf-a715c7a83d5f/global-pull-secret-syncer/0.log" Jul 08 19:45:43.379042 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:43.378991 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p5vml_428cf648-de10-4fc7-a55a-ffa7aad31575/konnectivity-agent/0.log" Jul 08 19:45:43.456780 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:43.456748 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-33.ec2.internal_802c451955599a360850d92f832cfcf3/haproxy/0.log" Jul 08 19:45:49.086074 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:49.086046 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xj977_458ac938-570c-402b-b44c-fd57cb3ef0ae/manager/0.log" Jul 08 19:45:49.115634 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:49.115613 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9t9fq_ca9c2331-85e4-4f8b-919c-e677298dd07d/kuadrant-console-plugin/0.log" Jul 08 19:45:49.364979 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:49.364893 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wn8f4_600443f6-e855-48ab-b5a0-411d96f04088/limitador/0.log" Jul 08 19:45:51.232294 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.232262 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca1fb528-0921-4044-90a7-134c42f800b9/alertmanager/0.log" Jul 08 19:45:51.256937 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.256910 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca1fb528-0921-4044-90a7-134c42f800b9/config-reloader/0.log" Jul 08 19:45:51.277478 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.277451 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca1fb528-0921-4044-90a7-134c42f800b9/kube-rbac-proxy-web/0.log" Jul 08 19:45:51.297674 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.297644 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca1fb528-0921-4044-90a7-134c42f800b9/kube-rbac-proxy/0.log" Jul 08 19:45:51.317446 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.317427 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca1fb528-0921-4044-90a7-134c42f800b9/kube-rbac-proxy-metric/0.log" Jul 08 19:45:51.341313 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.341295 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca1fb528-0921-4044-90a7-134c42f800b9/prom-label-proxy/0.log" Jul 08 19:45:51.360469 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.360438 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ca1fb528-0921-4044-90a7-134c42f800b9/init-config-reloader/0.log" Jul 08 19:45:51.436497 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.436415 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-sv47r_e3d39085-2b8b-4ca0-a91c-93912b9a847a/kube-state-metrics/0.log" Jul 08 19:45:51.453968 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.453945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-sv47r_e3d39085-2b8b-4ca0-a91c-93912b9a847a/kube-rbac-proxy-main/0.log" Jul 08 19:45:51.476163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.476102 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-sv47r_e3d39085-2b8b-4ca0-a91c-93912b9a847a/kube-rbac-proxy-self/0.log" Jul 08 19:45:51.501608 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.501583 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84c589b7bd-kwt7k_a445f661-6c1f-44d8-882b-61b6e9411392/metrics-server/0.log" Jul 08 19:45:51.525047 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.525025 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-x4g99_65154e7f-df89-4368-95ff-2b4147be5f9b/monitoring-plugin/0.log" Jul 08 19:45:51.558489 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.558459 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tr92_0e30e11f-9fec-4d54-8f79-01da25f8b3ae/node-exporter/0.log" Jul 08 19:45:51.577524 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.577504 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tr92_0e30e11f-9fec-4d54-8f79-01da25f8b3ae/kube-rbac-proxy/0.log" Jul 08 19:45:51.597171 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.597155 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tr92_0e30e11f-9fec-4d54-8f79-01da25f8b3ae/init-textfile/0.log" Jul 08 19:45:51.777636 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.777556 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-24sdz_9989c28e-0ee9-4684-a305-152f684d7a5d/kube-rbac-proxy-main/0.log" Jul 08 19:45:51.804925 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.804891 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-24sdz_9989c28e-0ee9-4684-a305-152f684d7a5d/kube-rbac-proxy-self/0.log" Jul 08 19:45:51.825689 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.825658 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-24sdz_9989c28e-0ee9-4684-a305-152f684d7a5d/openshift-state-metrics/0.log" Jul 08 19:45:51.861820 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.861792 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c24f2587-9388-47a9-a5ac-e52d66c77b82/prometheus/0.log" Jul 08 19:45:51.888495 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.888475 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c24f2587-9388-47a9-a5ac-e52d66c77b82/config-reloader/0.log" Jul 08 19:45:51.910804 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.910774 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c24f2587-9388-47a9-a5ac-e52d66c77b82/thanos-sidecar/0.log" Jul 08 19:45:51.930191 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.930168 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c24f2587-9388-47a9-a5ac-e52d66c77b82/kube-rbac-proxy-web/0.log" Jul 08 19:45:51.953773 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.953746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c24f2587-9388-47a9-a5ac-e52d66c77b82/kube-rbac-proxy/0.log" Jul 08 19:45:51.974487 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.974464 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c24f2587-9388-47a9-a5ac-e52d66c77b82/kube-rbac-proxy-thanos/0.log" Jul 08 19:45:51.993863 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:51.993835 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c24f2587-9388-47a9-a5ac-e52d66c77b82/init-config-reloader/0.log" Jul 08 19:45:52.023725 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.023706 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-9ccf9_a4f38d4c-09e7-427f-8f02-acf137525200/prometheus-operator/0.log" Jul 08 19:45:52.040096 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.040027 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-9ccf9_a4f38d4c-09e7-427f-8f02-acf137525200/kube-rbac-proxy/0.log" Jul 08 19:45:52.091956 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.091934 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b74f6f7dc-dfsvt_c3c36089-0bef-4f8f-a926-3f5a165e376c/telemeter-client/0.log" Jul 08 19:45:52.114513 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.114487 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b74f6f7dc-dfsvt_c3c36089-0bef-4f8f-a926-3f5a165e376c/reload/0.log" Jul 08 19:45:52.133530 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.133506 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b74f6f7dc-dfsvt_c3c36089-0bef-4f8f-a926-3f5a165e376c/kube-rbac-proxy/0.log" Jul 08 19:45:52.160097 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.160075 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77d5b8dbf5-c9k99_9031726c-a343-4f74-8d8d-ec4cfa5e9ec2/thanos-query/0.log" Jul 08 19:45:52.182684 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.182659 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77d5b8dbf5-c9k99_9031726c-a343-4f74-8d8d-ec4cfa5e9ec2/kube-rbac-proxy-web/0.log" Jul 08 19:45:52.202162 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.202138 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77d5b8dbf5-c9k99_9031726c-a343-4f74-8d8d-ec4cfa5e9ec2/kube-rbac-proxy/0.log" Jul 08 19:45:52.221431 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.221404 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77d5b8dbf5-c9k99_9031726c-a343-4f74-8d8d-ec4cfa5e9ec2/prom-label-proxy/0.log" Jul 08 19:45:52.241416 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.241388 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77d5b8dbf5-c9k99_9031726c-a343-4f74-8d8d-ec4cfa5e9ec2/kube-rbac-proxy-rules/0.log" Jul 08 19:45:52.261639 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:52.261618 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77d5b8dbf5-c9k99_9031726c-a343-4f74-8d8d-ec4cfa5e9ec2/kube-rbac-proxy-metrics/0.log" Jul 08 19:45:53.100378 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.100343 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5"] Jul 08 19:45:53.100825 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.100805 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca98722-cbf7-4fb7-b503-069aca87f924" containerName="test-curl-7660-140441388650832" Jul 08 19:45:53.100935 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.100827 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca98722-cbf7-4fb7-b503-069aca87f924" containerName="test-curl-7660-140441388650832" Jul 08 19:45:53.100935 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.100914 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca98722-cbf7-4fb7-b503-069aca87f924" containerName="test-curl-7660-140441388650832" Jul 08 19:45:53.103995 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.103975 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.106060 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.106037 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqj4d\"/\"openshift-service-ca.crt\"" Jul 08 19:45:53.106321 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.106305 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fqj4d\"/\"kube-root-ca.crt\"" Jul 08 19:45:53.107052 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.107034 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fqj4d\"/\"default-dockercfg-m444m\"" Jul 08 19:45:53.113262 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.113239 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5"] Jul 08 19:45:53.180707 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.180677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-podres\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.180707 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.180710 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-sys\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.180939 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.180752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-lib-modules\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.180939 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.180795 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddz6\" (UniqueName: \"kubernetes.io/projected/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-kube-api-access-kddz6\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.180939 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.180820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-proc\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-lib-modules\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282163 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddz6\" (UniqueName: \"kubernetes.io/projected/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-kube-api-access-kddz6\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282182 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-proc\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282263 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-podres\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-sys\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282313 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-lib-modules\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282331 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-proc\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282391 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-sys\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.282700 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.282396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-podres\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.290697 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.290674 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddz6\" (UniqueName: \"kubernetes.io/projected/6b3e14cc-8cfc-4ca3-bee1-6380801c1825-kube-api-access-kddz6\") pod \"perf-node-gather-daemonset-gcmf5\" (UID: \"6b3e14cc-8cfc-4ca3-bee1-6380801c1825\") " pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.414970 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.414945 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:53.550745 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:53.550719 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5"] Jul 08 19:45:53.553799 ip-10-0-140-33 kubenswrapper[2569]: W0708 19:45:53.553773 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b3e14cc_8cfc_4ca3_bee1_6380801c1825.slice/crio-3e0df4c9165dc3e9d10f0af4154d6ed54fca3c966eb64d5362e1774287e0dbcd WatchSource:0}: Error finding container 3e0df4c9165dc3e9d10f0af4154d6ed54fca3c966eb64d5362e1774287e0dbcd: Status 404 returned error can't find the container with id 3e0df4c9165dc3e9d10f0af4154d6ed54fca3c966eb64d5362e1774287e0dbcd Jul 08 19:45:54.280023 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:54.279990 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" event={"ID":"6b3e14cc-8cfc-4ca3-bee1-6380801c1825","Type":"ContainerStarted","Data":"4f346634da9c35a3b652a157a5915afa1dd498d056949562d2e96e6e8ac6242e"} Jul 08 19:45:54.280219 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:54.280049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" event={"ID":"6b3e14cc-8cfc-4ca3-bee1-6380801c1825","Type":"ContainerStarted","Data":"3e0df4c9165dc3e9d10f0af4154d6ed54fca3c966eb64d5362e1774287e0dbcd"} Jul 08 19:45:54.280219 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:54.280104 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:45:54.294410 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:54.294359 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" podStartSLOduration=1.294344388 podStartE2EDuration="1.294344388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:45:54.294020027 +0000 UTC m=+2499.733322664" watchObservedRunningTime="2026-07-08 19:45:54.294344388 +0000 UTC m=+2499.733647019" Jul 08 19:45:54.570645 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:54.570567 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-566b58fccd-47xhj_b6b22be7-2d22-44d6-88ff-512caa40c4d8/console/0.log" Jul 08 19:45:54.609018 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:54.608991 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-vw4jc_164eba15-890e-4e1b-a0e6-e7750d37db3d/download-server/0.log" Jul 08 19:45:56.252515 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:56.252471 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8k79_4f81f8d7-4135-4fc0-ab60-95701bca82d4/dns/0.log" Jul 08 19:45:56.271346 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:56.271323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8k79_4f81f8d7-4135-4fc0-ab60-95701bca82d4/kube-rbac-proxy/0.log" Jul 08 19:45:56.312973 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:56.312952 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l6jvn_e154b5fa-dac2-4f7f-bf8b-bdd8b79f8858/dns-node-resolver/0.log" Jul 08 19:45:56.894585 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:56.894530 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7rlf8_c108ae49-2a6c-44b5-a725-5d3a3b373713/node-ca/0.log" Jul 08 19:45:58.099123 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:58.099094 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdb786f-hkp72_9f18b0e9-db82-489a-b3c7-723eaa8e5cb8/kube-auth-proxy/0.log" Jul 08 19:45:59.080216 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:59.080182 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vj2f5_684f9a0f-00ac-47cf-884c-84d79e10e510/serve-healthcheck-canary/0.log" Jul 08 19:45:59.645502 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:59.645469 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4bg6_c89e967c-2e64-43c4-ba0e-ac8c221bb8f7/kube-rbac-proxy/0.log" Jul 08 19:45:59.665260 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:59.665231 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4bg6_c89e967c-2e64-43c4-ba0e-ac8c221bb8f7/exporter/0.log" Jul 08 19:45:59.686086 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:45:59.686061 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4bg6_c89e967c-2e64-43c4-ba0e-ac8c221bb8f7/extractor/0.log" Jul 08 19:46:00.296641 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:00.296615 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fqj4d/perf-node-gather-daemonset-gcmf5" Jul 08 19:46:02.639051 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:02.639019 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-84g49_799e5671-ff3e-407b-a3ce-c9268e717ae1/manager/0.log" Jul 08 19:46:02.659237 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:02.659212 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-994cc9f69-84g49_799e5671-ff3e-407b-a3ce-c9268e717ae1/copy-manifests/0.log" Jul 08 19:46:03.936654 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:03.936623 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69dfbfd578-x7lxw_7b0398c0-28cd-4bbe-a107-6f9e33ad87d9/manager/0.log" Jul 08 19:46:03.961325 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:03.961295 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hn6hg_8808ab32-7df7-45ea-9abb-924ab62e8728/openshift-lws-operator/0.log" Jul 08 19:46:04.351295 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:04.351215 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdb786f-hkp72_9f18b0e9-db82-489a-b3c7-723eaa8e5cb8/kube-auth-proxy/0.log" Jul 08 19:46:05.900324 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:05.900294 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdb786f-hkp72_9f18b0e9-db82-489a-b3c7-723eaa8e5cb8/kube-auth-proxy/0.log" Jul 08 19:46:10.272842 ip-10-0-140-33 kubenswrapper[2569]: I0708 19:46:10.272814 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4jqnv_a30ca408-d43a-4fc1-bb33-92de7d6fa99d/kube-multus/0.log"