Jul 06 23:48:55.253832 ip-10-0-139-100 systemd[1]: Starting Kubernetes Kubelet... Jul 06 23:48:55.588938 ip-10-0-139-100 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:48:55.588938 ip-10-0-139-100 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 23:48:55.588938 ip-10-0-139-100 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:48:55.588938 ip-10-0-139-100 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 23:48:55.588938 ip-10-0-139-100 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:48:55.590379 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.590303 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 23:48:55.596115 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596100 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:48:55.596115 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596116 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596120 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596123 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596126 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596129 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596132 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596134 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596137 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596140 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596142 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596145 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596148 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596150 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596152 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596155 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596157 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596160 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596163 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596165 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596168 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:48:55.596182 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596170 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596173 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596175 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596183 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596187 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596190 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596192 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596195 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596198 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596200 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596203 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596205 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596208 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596210 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596213 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596217 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596220 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596223 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596226 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:48:55.596666 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596229 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596231 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596233 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596236 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596239 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596241 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596244 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596246 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596248 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596251 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596253 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596255 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596258 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596260 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596263 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596266 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596269 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596274 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596277 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:48:55.597124 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596279 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596282 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596286 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596289 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596292 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596295 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596297 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596300 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596302 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596304 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596307 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596309 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596311 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596314 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596317 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596319 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596322 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596324 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596327 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596329 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:48:55.597593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596332 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596334 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596337 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596339 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596341 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596344 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596346 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596348 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596351 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596354 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596357 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596359 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596362 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596364 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596366 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596369 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596371 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596374 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596376 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.596378 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:48:55.598077 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597080 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597087 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597091 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597094 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597096 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597099 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597101 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597104 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597108 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597112 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597115 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597118 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597121 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597123 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597126 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597128 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597131 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597134 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597137 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:48:55.598623 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597139 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597143 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597146 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597149 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597152 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597154 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597157 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597159 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597162 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597164 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597166 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597169 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597171 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597175 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597179 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597182 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597185 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597187 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597189 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:48:55.599219 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597192 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597194 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597197 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597199 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597201 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597204 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597208 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597210 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597212 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597216 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597219 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597222 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597225 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597227 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597231 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597234 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597236 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597239 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597242 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:48:55.599700 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597244 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597246 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597249 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597251 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597254 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597256 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597259 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597261 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597264 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597266 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597268 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597271 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597273 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597275 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597278 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597280 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597283 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597285 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597287 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597290 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:48:55.600162 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597293 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597296 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597299 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597301 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597304 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597306 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597308 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597312 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597315 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597318 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597320 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597323 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597325 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597327 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597330 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597332 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597335 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597338 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597340 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597343 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:48:55.600653 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597345 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597347 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597758 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597763 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597767 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597770 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597773 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597777 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597781 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597784 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597786 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597789 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597792 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597794 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597797 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597800 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597802 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597804 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597807 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:48:55.601129 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597812 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597815 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597818 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597821 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597823 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597825 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597828 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597830 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597833 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597835 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597838 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597840 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597843 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597845 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597847 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597849 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597852 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597855 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597857 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597859 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:48:55.601593 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597862 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597864 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597867 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597869 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597873 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597876 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597880 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597883 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597885 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597887 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597890 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597892 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597897 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597899 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597902 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597904 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597907 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597909 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597911 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:48:55.602065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597914 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597916 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597920 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597922 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597925 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597927 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597930 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597932 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597935 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597938 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597940 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597943 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597945 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597948 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597950 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597953 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597955 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597958 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597960 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:48:55.602573 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597965 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597968 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597971 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597974 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597976 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597979 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597985 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597987 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597990 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597992 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597995 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.597997 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598000 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598003 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598005 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598007 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598010 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598012 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598015 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598017 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:48:55.603020 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598020 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598022 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598024 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598027 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598088 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598094 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598099 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598104 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598108 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598111 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598115 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598119 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598121 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598124 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598128 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598131 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598137 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598141 2564 flags.go:64] FLAG: --cgroup-root="" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598147 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598151 2564 flags.go:64] FLAG: --client-ca-file="" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598154 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598159 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598167 2564 flags.go:64] FLAG: --cluster-domain="" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598170 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 23:48:55.603488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598175 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598178 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598181 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598184 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598187 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598190 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598193 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598196 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598199 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598202 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598205 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598207 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598211 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598214 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598217 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598219 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598222 2564 flags.go:64] FLAG: --enable-server="true" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598225 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598228 2564 flags.go:64] FLAG: --event-burst="100" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598231 2564 flags.go:64] FLAG: --event-qps="50" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598234 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598237 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598240 2564 flags.go:64] FLAG: --eviction-hard="" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598244 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 23:48:55.604105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598247 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598250 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598254 2564 flags.go:64] FLAG: --eviction-soft="" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598257 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598260 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598263 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598266 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598269 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598271 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598274 2564 flags.go:64] FLAG: --feature-gates="" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598277 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598281 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598283 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598286 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598289 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598292 2564 flags.go:64] FLAG: --help="false" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598295 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598298 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598301 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598303 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598306 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598309 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598312 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 23:48:55.604704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598315 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598317 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598320 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598323 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598326 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598329 2564 flags.go:64] FLAG: --kube-reserved="" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598331 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598334 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598337 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598340 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598343 2564 flags.go:64] FLAG: --lock-file="" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598348 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598351 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598354 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598358 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598361 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598364 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598366 2564 flags.go:64] FLAG: --logging-format="text" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598369 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598372 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598375 2564 flags.go:64] FLAG: --manifest-url="" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598377 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598381 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598384 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598388 2564 flags.go:64] FLAG: --max-pods="110" Jul 06 23:48:55.605279 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598390 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598393 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598396 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598399 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598402 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598404 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598407 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598413 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598416 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598419 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598422 2564 flags.go:64] FLAG: --pod-cidr="" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598425 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598430 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598432 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598435 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598438 2564 flags.go:64] FLAG: --port="10250" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598441 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598444 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0add656f2b1ab470f" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598447 2564 flags.go:64] FLAG: --qos-reserved="" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598451 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598454 2564 flags.go:64] FLAG: --register-node="true" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598456 2564 flags.go:64] FLAG: --register-with-taints="" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598460 2564 flags.go:64] FLAG: --registry-burst="10" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598463 2564 flags.go:64] FLAG: --registry-qps="5" Jul 06 23:48:55.605894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598466 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598468 2564 flags.go:64] FLAG: --reserved-memory="" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598471 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598474 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598477 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598480 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598482 2564 flags.go:64] FLAG: --runonce="false" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598485 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598488 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598491 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598511 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598514 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598517 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598519 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598523 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598525 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598528 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598531 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598534 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598536 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598539 2564 flags.go:64] FLAG: --system-cgroups="" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598542 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598547 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598550 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598552 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 23:48:55.606474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598556 2564 flags.go:64] FLAG: --tls-min-version="" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598561 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598565 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598567 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598570 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598573 2564 flags.go:64] FLAG: --v="2" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598576 2564 flags.go:64] FLAG: --version="false" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598580 2564 flags.go:64] FLAG: --vmodule="" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598583 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.598586 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598707 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598712 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598715 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598718 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598721 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598723 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598726 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598729 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598733 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598737 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598739 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598742 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:48:55.607109 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598745 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598748 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598750 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598753 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598755 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598758 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598760 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598763 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598765 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598767 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598770 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598774 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598778 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598780 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598783 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598785 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598787 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598790 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598793 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598796 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:48:55.607642 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598798 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598800 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598803 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598805 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598808 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598810 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598813 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598815 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598818 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598820 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598823 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598825 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598827 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598830 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598832 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598835 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598837 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598839 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598842 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598845 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:48:55.608127 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598847 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598849 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598852 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598856 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598859 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598862 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598865 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598867 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598870 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598872 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598876 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598879 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598882 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598884 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598887 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598889 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598892 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598894 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598897 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:48:55.608624 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598899 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598902 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598904 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598906 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598909 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598911 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598914 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598916 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598920 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598923 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598926 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598928 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598931 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598934 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598937 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598940 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598943 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598947 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598949 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598953 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:48:55.609094 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598955 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598958 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598960 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598963 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598965 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598967 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598970 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.598972 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.599568 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.606408 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.606421 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.607061 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.607068 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607131 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607136 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:48:55.609581 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607139 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607142 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607144 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607147 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607150 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607153 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607155 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607158 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607161 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607163 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607166 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607168 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607171 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607174 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607178 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607183 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607192 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607195 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607197 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:48:55.609953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607199 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607202 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607205 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607207 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607210 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607212 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607214 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607217 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607219 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607222 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607224 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607226 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607229 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607231 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607234 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607236 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607238 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607241 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607244 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:48:55.610403 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607246 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607248 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607251 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607253 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607256 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607264 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607267 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607269 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607276 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607279 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607281 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607289 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607291 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607294 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607296 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607299 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607301 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607303 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607306 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:48:55.610865 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607308 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607310 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607313 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607315 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607318 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607320 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607323 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607325 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607327 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607330 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607332 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607334 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607337 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607339 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607341 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607344 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607346 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607348 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607351 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607353 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607356 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:48:55.611315 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607359 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607361 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607365 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607368 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607377 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607379 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607382 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607385 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607387 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607390 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607392 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607396 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607398 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607401 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607403 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607406 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607409 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607411 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:48:55.611890 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607413 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.607419 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607565 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607570 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607572 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607575 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607577 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607581 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607585 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607589 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607592 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607595 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607597 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607600 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:48:55.612323 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607603 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607605 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607608 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607610 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607612 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607620 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607623 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607625 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607628 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607630 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607633 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607635 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607637 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607640 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607642 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607645 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607647 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607650 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607653 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607655 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:48:55.612678 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607658 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607660 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607662 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607665 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607667 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607669 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607672 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607674 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607676 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607679 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607681 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607684 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607688 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607691 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607694 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607696 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607699 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607702 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607713 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:48:55.613142 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607715 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607718 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607720 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607723 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607725 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607727 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607730 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607732 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607735 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607737 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607739 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607742 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607744 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607747 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607749 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607752 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607754 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607756 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607759 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607761 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:48:55.613596 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607763 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607766 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607768 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607770 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607773 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607775 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607778 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607783 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607785 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607788 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607790 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607793 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607802 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607804 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607807 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607809 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607812 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607814 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607816 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607819 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:48:55.614065 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607821 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607824 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607826 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607828 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607831 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607833 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607835 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:55.607838 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.607842 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.607942 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.609971 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.611156 2564 server.go:1019] "Starting client certificate rotation" Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.611258 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.611288 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:48:55.614542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.613605 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:48:55.619666 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.619626 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:48:55.628603 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.628586 2564 log.go:25] "Validated CRI v1 runtime API" Jul 06 23:48:55.632346 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.632328 2564 log.go:25] "Validated CRI v1 image API" Jul 06 23:48:55.633180 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.633161 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 23:48:55.635341 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.635321 2564 fs.go:135] Filesystem UUIDs: map[79ffdcf2-4959-4647-92e7-1f64eb50a68c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 acbbe0ca-3e36-4dea-b890-5598be2a09c7:/dev/nvme0n1p3] Jul 06 23:48:55.635409 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.635340 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 23:48:55.640530 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.640371 2564 manager.go:216] Machine: {Timestamp:2026-07-06 23:48:55.639255506 +0000 UTC m=+0.289677464 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097777 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b6cf14533efdc6c2f623b6127d27a SystemUUID:ec2b6cf1-4533-efdc-6c2f-623b6127d27a BootID:c7ea419d-21ce-4d09-9c16-4639f3065087 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:27:05:fd:61:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:27:05:fd:61:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:48:8d:b1:8a:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 23:48:55.640530 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.640523 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 23:48:55.640645 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.640631 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 23:48:55.642598 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.642584 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:48:55.643127 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.643102 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 23:48:55.643255 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.643131 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 23:48:55.643767 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.643758 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 23:48:55.643804 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.643771 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 23:48:55.643804 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.643783 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:48:55.644320 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.644311 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:48:55.645631 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.645620 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 23:48:55.649846 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.649835 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:48:55.649957 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.649947 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 23:48:55.649988 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.649978 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 06 23:48:55.650018 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.649992 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 23:48:55.650018 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.650011 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 23:48:55.650067 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.650023 2564 kubelet.go:398] "Adding apiserver pod source" Jul 06 23:48:55.650067 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.650041 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 23:48:55.650540 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.650524 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:48:55.650599 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.650552 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:48:55.652375 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.652337 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 23:48:55.654611 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.654599 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 23:48:55.654657 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.654628 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 23:48:55.655980 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.655967 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 23:48:55.656037 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.655985 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 23:48:55.656037 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.655992 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 23:48:55.656037 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.655999 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 23:48:55.656037 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656011 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 23:48:55.656037 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656020 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 23:48:55.656037 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656037 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 23:48:55.656233 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656047 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 23:48:55.656233 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656066 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 23:48:55.656233 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656071 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 23:48:55.656233 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656081 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 23:48:55.656233 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656086 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 23:48:55.656985 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.656902 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 23:48:55.657129 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.657104 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 23:48:55.660845 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.660826 2564 server.go:1268] "Started kubelet" Jul 06 23:48:55.660943 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.660862 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 23:48:55.660943 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.660872 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 23:48:55.661022 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.660957 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 23:48:55.662275 ip-10-0-139-100 systemd[1]: Started Kubernetes Kubelet. Jul 06 23:48:55.663360 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.663184 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 06 23:48:55.664866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.664808 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 23:48:55.664957 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.664896 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 23:48:55.665129 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.665109 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 23:48:55.665398 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.665385 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 23:48:55.666115 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.666095 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rnklg" Jul 06 23:48:55.666315 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.666293 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 23:48:55.669398 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.668679 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-100.ec2.internal.18bfd83d64fa6c11 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-100.ec2.internal,UID:ip-10-0-139-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-100.ec2.internal,},FirstTimestamp:2026-07-06 23:48:55.660743697 +0000 UTC m=+0.311165638,LastTimestamp:2026-07-06 23:48:55.660743697 +0000 UTC m=+0.311165638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-100.ec2.internal,}" Jul 06 23:48:55.670448 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.670430 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 23:48:55.670545 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.670447 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 23:48:55.670613 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.670565 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:55.671032 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.671017 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 23:48:55.671104 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.671053 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 23:48:55.671104 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.671020 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 23:48:55.671194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.671171 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 06 23:48:55.671194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.671179 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 06 23:48:55.671855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.671839 2564 factory.go:55] Registering systemd factory Jul 06 23:48:55.671855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.671856 2564 factory.go:223] Registration of the systemd container factory successfully Jul 06 23:48:55.672055 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.672040 2564 factory.go:153] Registering CRI-O factory Jul 06 23:48:55.672114 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.672059 2564 factory.go:223] Registration of the crio container factory successfully Jul 06 23:48:55.672162 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.672145 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 23:48:55.672211 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.672171 2564 factory.go:103] Registering Raw factory Jul 06 23:48:55.672211 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.672196 2564 manager.go:1196] Started watching for new ooms in manager Jul 06 23:48:55.672688 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.672667 2564 manager.go:349] Starting recovery of all containers Jul 06 23:48:55.674164 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.674120 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rnklg" Jul 06 23:48:55.681820 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.681794 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 23:48:55.681935 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.681912 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 23:48:55.683060 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.683042 2564 manager.go:354] Recovery completed Jul 06 23:48:55.687530 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.687518 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:48:55.689245 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.689232 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:48:55.689292 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.689259 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:48:55.689292 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.689269 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:48:55.689785 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.689771 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 23:48:55.689785 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.689781 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 23:48:55.689922 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.689792 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:48:55.689922 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.689795 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:48:55.690471 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.690458 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:48:55.690564 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.690483 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:48:55.690564 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.690514 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:48:55.691799 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.691783 2564 policy_none.go:49] "None policy: Start" Jul 06 23:48:55.691799 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.691798 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 23:48:55.691883 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.691808 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 23:48:55.693758 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.693748 2564 policy_none.go:47] "Start" Jul 06 23:48:55.720061 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.720040 2564 manager.go:337] "Starting Device Plugin manager" Jul 06 23:48:55.720151 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.720111 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 23:48:55.720151 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.720127 2564 server.go:87] "Starting device plugin registration server" Jul 06 23:48:55.720346 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.720332 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 23:48:55.720403 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.720346 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 23:48:55.720456 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.720438 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 23:48:55.720550 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.720537 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 23:48:55.720598 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.720550 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 23:48:55.721023 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.721004 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 23:48:55.721098 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.721044 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:55.797734 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.797707 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 23:48:55.799854 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.798936 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 23:48:55.799854 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.798965 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 23:48:55.799854 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.798985 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 23:48:55.799854 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.799013 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 23:48:55.801485 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.801464 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:48:55.821441 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.821421 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:48:55.822183 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.822163 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:48:55.822262 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.822191 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:48:55.822262 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.822200 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:48:55.822262 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.822222 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.828142 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.828126 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.828196 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.828144 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-100.ec2.internal\": node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:55.840705 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.840657 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:55.899598 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.899572 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"] Jul 06 23:48:55.899705 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.899640 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:48:55.901065 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.901050 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:48:55.901120 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.901078 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:48:55.901120 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.901089 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:48:55.903283 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903271 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:48:55.903434 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903419 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.903488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903449 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:48:55.903964 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903949 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:48:55.903964 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903961 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:48:55.904048 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903975 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:48:55.904048 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903981 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:48:55.904048 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903987 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:48:55.904048 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.903991 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:48:55.906180 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.906167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.906241 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.906191 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:48:55.906831 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.906814 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:48:55.906925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.906842 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:48:55.906925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.906852 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:48:55.928047 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.928031 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.932418 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.932402 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.941612 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:55.941596 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:55.972700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.972678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8ecbc7fda6a812262efefe3033234e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"7f8ecbc7fda6a812262efefe3033234e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.972770 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.972703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/319f82ace8919ccdff136f05eb05d345-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"319f82ace8919ccdff136f05eb05d345\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 06 23:48:55.972770 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:55.972722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f8ecbc7fda6a812262efefe3033234e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"7f8ecbc7fda6a812262efefe3033234e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.041941 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.041921 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.073248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.073220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f8ecbc7fda6a812262efefe3033234e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"7f8ecbc7fda6a812262efefe3033234e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.073309 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.073253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8ecbc7fda6a812262efefe3033234e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"7f8ecbc7fda6a812262efefe3033234e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.073309 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.073271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/319f82ace8919ccdff136f05eb05d345-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"319f82ace8919ccdff136f05eb05d345\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.073309 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.073308 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/319f82ace8919ccdff136f05eb05d345-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"319f82ace8919ccdff136f05eb05d345\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.073398 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.073324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f8ecbc7fda6a812262efefe3033234e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"7f8ecbc7fda6a812262efefe3033234e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.073398 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.073334 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f8ecbc7fda6a812262efefe3033234e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"7f8ecbc7fda6a812262efefe3033234e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.142560 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.142486 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.230185 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.230153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.235289 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.235170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 06 23:48:56.243392 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.243373 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.343867 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.343835 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.444433 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.444366 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.545097 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.545066 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.612986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.612961 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 23:48:56.613712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.613085 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:48:56.645241 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.645213 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.679555 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.679521 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 23:43:55 +0000 UTC" deadline="2028-02-15 07:29:55.28516764 +0000 UTC" Jul 06 23:48:56.679555 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.679553 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14119h40m58.605617386s" Jul 06 23:48:56.686882 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.686863 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:48:56.728593 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.728535 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:48:56.745412 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.745382 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.788189 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:56.788151 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319f82ace8919ccdff136f05eb05d345.slice/crio-398ce176248e7011141a1dbe015dacc325f4d80fcc463f0b3fc14132dd156183 WatchSource:0}: Error finding container 398ce176248e7011141a1dbe015dacc325f4d80fcc463f0b3fc14132dd156183: Status 404 returned error can't find the container with id 398ce176248e7011141a1dbe015dacc325f4d80fcc463f0b3fc14132dd156183 Jul 06 23:48:56.792680 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.792660 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:48:56.798442 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:56.798421 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8ecbc7fda6a812262efefe3033234e.slice/crio-e666bf1d2b33f145cd6e9b988f011718ef9781e7a315a92a950e509aed0ad6ee WatchSource:0}: Error finding container e666bf1d2b33f145cd6e9b988f011718ef9781e7a315a92a950e509aed0ad6ee: Status 404 returned error can't find the container with id e666bf1d2b33f145cd6e9b988f011718ef9781e7a315a92a950e509aed0ad6ee Jul 06 23:48:56.802315 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.802263 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"7f8ecbc7fda6a812262efefe3033234e","Type":"ContainerStarted","Data":"e666bf1d2b33f145cd6e9b988f011718ef9781e7a315a92a950e509aed0ad6ee"} Jul 06 23:48:56.803378 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:56.803355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"319f82ace8919ccdff136f05eb05d345","Type":"ContainerStarted","Data":"398ce176248e7011141a1dbe015dacc325f4d80fcc463f0b3fc14132dd156183"} Jul 06 23:48:56.847396 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.847371 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:56.948096 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:56.948076 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:57.049214 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.049155 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:57.149741 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.149710 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 06 23:48:57.207850 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.207821 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:48:57.270715 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.270680 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 06 23:48:57.281599 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.281567 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:48:57.282851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.282689 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 06 23:48:57.292537 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.292515 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:48:57.569219 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.569184 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:48:57.651513 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.651437 2564 apiserver.go:52] "Watching apiserver" Jul 06 23:48:57.659585 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.659555 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 23:48:57.659963 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.659930 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d","openshift-cluster-node-tuning-operator/tuned-jpt9b","openshift-image-registry/node-ca-s9ds8","openshift-multus/network-metrics-daemon-5k58z","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal","openshift-dns/node-resolver-5kssz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","openshift-multus/multus-additional-cni-plugins-ttrjd","openshift-multus/multus-kqprl","openshift-network-diagnostics/network-check-target-mptrb","openshift-network-operator/iptables-alerter-8mxn8","openshift-ovn-kubernetes/ovnkube-node-bjgzw","kube-system/global-pull-secret-syncer-zlnnn","kube-system/konnectivity-agent-s5wl5"] Jul 06 23:48:57.662798 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.662779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:48:57.662922 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.662869 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:48:57.665125 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.665097 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.667473 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.667450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.668207 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.668188 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 23:48:57.668436 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.668422 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:48:57.668680 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.668659 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 23:48:57.668761 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.668666 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ck7zt\"" Jul 06 23:48:57.669474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.669450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 23:48:57.669564 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.669494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 23:48:57.669564 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.669546 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ndxkx\"" Jul 06 23:48:57.669774 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.669753 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 23:48:57.670031 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.670014 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:57.670103 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.670086 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:48:57.671047 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.670997 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 23:48:57.672326 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.672306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:57.674470 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.674448 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 23:48:57.674587 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.674500 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c7rnr\"" Jul 06 23:48:57.674655 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.674450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 23:48:57.674805 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.674791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.677164 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.677144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.677355 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.677333 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 23:48:57.677588 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.677523 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bd996\"" Jul 06 23:48:57.677767 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.677748 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 23:48:57.679231 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.679206 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 23:48:57.679510 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.679488 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dhrjm\"" Jul 06 23:48:57.679702 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.679688 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 23:48:57.679832 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.679821 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 23:48:57.680557 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.680541 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 23:48:57.681646 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.681626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.681744 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.681692 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:48:57.682247 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.682247 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05e886f2-5943-4198-afdd-3ae2ea8d90b7-iptables-alerter-script\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.682546 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05e886f2-5943-4198-afdd-3ae2ea8d90b7-host-slash\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.682631 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hh7\" (UniqueName: \"kubernetes.io/projected/05e886f2-5943-4198-afdd-3ae2ea8d90b7-kube-api-access-d9hh7\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.682691 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ef88259-880c-4c27-a7c5-500f8b1f2698-serviceca\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.682749 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:48:57.682749 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ef88259-880c-4c27-a7c5-500f8b1f2698-host\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.682851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvrh\" (UniqueName: \"kubernetes.io/projected/1ef88259-880c-4c27-a7c5-500f8b1f2698-kube-api-access-nlvrh\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.682851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:57.682851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682829 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:48:57.683002 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bmz\" (UniqueName: \"kubernetes.io/projected/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-kube-api-access-r7bmz\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:57.683057 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.682990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5352efee-cf3f-463a-82ff-8e9cb779e171-agent-certs\") pod \"konnectivity-agent-s5wl5\" (UID: \"5352efee-cf3f-463a-82ff-8e9cb779e171\") " pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:57.683057 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.683034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5352efee-cf3f-463a-82ff-8e9cb779e171-konnectivity-ca\") pod \"konnectivity-agent-s5wl5\" (UID: \"5352efee-cf3f-463a-82ff-8e9cb779e171\") " pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:57.684844 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.684819 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hjz5k\"" Jul 06 23:48:57.685138 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.685120 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:48:57.685615 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.685563 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 23:48:57.687314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.687295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.689970 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.689946 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 23:48:57.690096 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.689987 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 23:48:57.690151 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.690094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 23:48:57.690448 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.690285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pb266\"" Jul 06 23:48:57.691307 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.691289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.693277 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.693105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 23:48:57.693277 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.693243 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 23:48:57.693440 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.693400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8mbt\"" Jul 06 23:48:57.693523 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.693443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.695200 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.695181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 23:48:57.695573 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.695554 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 23:48:57.695871 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.695854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 23:48:57.696082 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.696038 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 23:48:57.696479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.696459 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t59k7\"" Jul 06 23:48:57.696605 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.696458 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 23:48:57.696605 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.696498 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 23:48:57.704072 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.704052 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wswqj" Jul 06 23:48:57.713445 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.713424 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wswqj" Jul 06 23:48:57.772307 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.772284 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 23:48:57.784196 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5352efee-cf3f-463a-82ff-8e9cb779e171-konnectivity-ca\") pod \"konnectivity-agent-s5wl5\" (UID: \"5352efee-cf3f-463a-82ff-8e9cb779e171\") " pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:57.784314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-device-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.784314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-os-release\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.784314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-ovnkube-config\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.784314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-sys-fs\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.784314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2afa112-61f1-41f2-9d75-c3647002d54b-cni-binary-copy\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.784609 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.784609 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.784609 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls85x\" (UniqueName: \"kubernetes.io/projected/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-kube-api-access-ls85x\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.784858 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784837 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5352efee-cf3f-463a-82ff-8e9cb779e171-konnectivity-ca\") pod \"konnectivity-agent-s5wl5\" (UID: \"5352efee-cf3f-463a-82ff-8e9cb779e171\") " pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:57.784921 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05e886f2-5943-4198-afdd-3ae2ea8d90b7-host-slash\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.784976 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-registration-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.784976 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-system-cni-dir\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.785052 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785010 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05e886f2-5943-4198-afdd-3ae2ea8d90b7-host-slash\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.785052 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.784983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-ovn\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.785139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-node-log\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.785139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-sys\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.785139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ef88259-880c-4c27-a7c5-500f8b1f2698-host\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.785139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-cnibin\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.785139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-cni-multus\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-daemon-config\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-hostroot\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785190 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ef88259-880c-4c27-a7c5-500f8b1f2698-host\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/654e5a71-df1b-4548-a477-fd86247ce561-ovn-node-metrics-cert\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-kubernetes\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.785340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysctl-conf\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p42f\" (UniqueName: \"kubernetes.io/projected/13605fab-a33b-41d6-8f26-0c2defe642af-kube-api-access-4p42f\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-tuned\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05e886f2-5943-4198-afdd-3ae2ea8d90b7-iptables-alerter-script\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvrh\" (UniqueName: \"kubernetes.io/projected/1ef88259-880c-4c27-a7c5-500f8b1f2698-kube-api-access-nlvrh\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-socket-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-log-socket\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-cni-netd\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlm7n\" (UniqueName: \"kubernetes.io/projected/654e5a71-df1b-4548-a477-fd86247ce561-kube-api-access-vlm7n\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysconfig\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysctl-d\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-netns\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.785809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-cni-bin\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-modprobe-d\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4wcx\" (UniqueName: \"kubernetes.io/projected/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-kube-api-access-k4wcx\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ef88259-880c-4c27-a7c5-500f8b1f2698-serviceca\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.785971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-os-release\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-k8s-cni-cncf-io\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-systemd\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-systemd\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-run\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-tmp\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786138 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05e886f2-5943-4198-afdd-3ae2ea8d90b7-iptables-alerter-script\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-multus-certs\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hh7\" (UniqueName: \"kubernetes.io/projected/05e886f2-5943-4198-afdd-3ae2ea8d90b7-kube-api-access-d9hh7\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-env-overrides\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-var-lib-kubelet\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.786437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4wb\" (UniqueName: \"kubernetes.io/projected/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-kube-api-access-6m4wb\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-kubelet\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tm4w\" (UniqueName: \"kubernetes.io/projected/d2afa112-61f1-41f2-9d75-c3647002d54b-kube-api-access-6tm4w\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786389 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ef88259-880c-4c27-a7c5-500f8b1f2698-serviceca\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-cnibin\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.786524 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.786640 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs podName:1267abdc-57d6-410e-9b07-6fa8dfdd5ac7 nodeName:}" failed. No retries permitted until 2026-07-06 23:48:58.286553925 +0000 UTC m=+2.936975852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs") pod "network-metrics-daemon-5k58z" (UID: "1267abdc-57d6-410e-9b07-6fa8dfdd5ac7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-run-netns\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-cni-bin\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-ovnkube-script-lib\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bmz\" (UniqueName: \"kubernetes.io/projected/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-kube-api-access-r7bmz\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.787194 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-system-cni-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-socket-dir-parent\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.786914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-etc-kubernetes\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-var-lib-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-hosts-file\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-tmp-dir\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5352efee-cf3f-463a-82ff-8e9cb779e171-agent-certs\") pod \"konnectivity-agent-s5wl5\" (UID: \"5352efee-cf3f-463a-82ff-8e9cb779e171\") " pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-cni-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-conf-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-kubelet-config\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787861 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-dbus\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-lib-modules\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.787984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.787967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-host\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.788597 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.788000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-kubelet\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.788597 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.788050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-systemd-units\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.788597 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.788077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-slash\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.788597 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.788109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-etc-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.793490 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.793472 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:48:57.793490 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.793495 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:48:57.793490 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.793510 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sbf9s for pod openshift-network-diagnostics/network-check-target-mptrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:48:57.793704 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.793634 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s podName:edba5765-8d28-4002-9d52-de6a671e6178 nodeName:}" failed. No retries permitted until 2026-07-06 23:48:58.293558571 +0000 UTC m=+2.943980520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sbf9s" (UniqueName: "kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s") pod "network-check-target-mptrb" (UID: "edba5765-8d28-4002-9d52-de6a671e6178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:48:57.795800 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.795776 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvrh\" (UniqueName: \"kubernetes.io/projected/1ef88259-880c-4c27-a7c5-500f8b1f2698-kube-api-access-nlvrh\") pod \"node-ca-s9ds8\" (UID: \"1ef88259-880c-4c27-a7c5-500f8b1f2698\") " pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.796055 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.796034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bmz\" (UniqueName: \"kubernetes.io/projected/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-kube-api-access-r7bmz\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:57.796129 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.796114 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hh7\" (UniqueName: \"kubernetes.io/projected/05e886f2-5943-4198-afdd-3ae2ea8d90b7-kube-api-access-d9hh7\") pod \"iptables-alerter-8mxn8\" (UID: \"05e886f2-5943-4198-afdd-3ae2ea8d90b7\") " pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.796744 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.796723 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5352efee-cf3f-463a-82ff-8e9cb779e171-agent-certs\") pod \"konnectivity-agent-s5wl5\" (UID: \"5352efee-cf3f-463a-82ff-8e9cb779e171\") " pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:57.889388 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-registration-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.889388 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-system-cni-dir\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.889388 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-ovn\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.889388 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-node-log\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-sys\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-cnibin\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889445 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-system-cni-dir\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889448 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-ovn\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-cni-multus\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889499 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-sys\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889500 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-cnibin\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-daemon-config\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889547 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-cni-multus\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889476 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-registration-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889533 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-node-log\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-hostroot\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889617 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-hostroot\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/654e5a71-df1b-4548-a477-fd86247ce561-ovn-node-metrics-cert\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889657 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.889700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889681 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-kubernetes\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysctl-conf\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p42f\" (UniqueName: \"kubernetes.io/projected/13605fab-a33b-41d6-8f26-0c2defe642af-kube-api-access-4p42f\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-tuned\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-socket-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-log-socket\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-cni-netd\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlm7n\" (UniqueName: \"kubernetes.io/projected/654e5a71-df1b-4548-a477-fd86247ce561-kube-api-access-vlm7n\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysconfig\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysctl-d\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-netns\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-cni-bin\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.889991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890065 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-log-socket\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890103 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-daemon-config\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-modprobe-d\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890147 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.890488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4wcx\" (UniqueName: \"kubernetes.io/projected/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-kube-api-access-k4wcx\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890234 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-modprobe-d\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890302 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-os-release\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-kubernetes\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890423 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-socket-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-os-release\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-k8s-cni-cncf-io\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890469 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890488 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-systemd\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-systemd\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-run\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890564 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysctl-conf\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890586 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-run-systemd\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-tmp\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890631 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-cni-netd\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-multus-certs\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.891367 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-env-overrides\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890554 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysconfig\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-var-lib-kubelet\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890711 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-run\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890672 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-multus-certs\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890730 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-sysctl-d\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890755 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-var-lib-kubelet\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890771 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-systemd\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890807 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-netns\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890818 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-cni-bin\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890862 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-run-k8s-cni-cncf-io\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4wb\" (UniqueName: \"kubernetes.io/projected/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-kube-api-access-6m4wb\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-kubelet\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tm4w\" (UniqueName: \"kubernetes.io/projected/d2afa112-61f1-41f2-9d75-c3647002d54b-kube-api-access-6tm4w\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890991 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-kubelet\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.890996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-cnibin\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.892318 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891038 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-cnibin\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-run-netns\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-cni-bin\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891115 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-env-overrides\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.891137 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891171 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891171 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-run-netns\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:57.891211 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret podName:1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606 nodeName:}" failed. No retries permitted until 2026-07-06 23:48:58.391193959 +0000 UTC m=+3.041615888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret") pod "global-pull-secret-syncer-zlnnn" (UID: "1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891211 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-cni-bin\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-ovnkube-script-lib\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-system-cni-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-socket-dir-parent\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-etc-kubernetes\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-var-lib-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891485 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-system-cni-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891478 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-var-lib-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891529 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-etc-kubernetes\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-hosts-file\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-tmp-dir\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891632 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-socket-dir-parent\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-cni-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891659 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13605fab-a33b-41d6-8f26-0c2defe642af-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891619 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-conf-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-hosts-file\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-kubelet-config\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891737 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-conf-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-dbus\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891755 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-multus-cni-dir\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-lib-modules\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891792 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-kubelet-config\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-host\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.893855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891822 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-ovnkube-script-lib\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-kubelet\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-systemd-units\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891869 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2afa112-61f1-41f2-9d75-c3647002d54b-host-var-lib-kubelet\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891891 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-lib-modules\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891897 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-dbus\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-slash\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891908 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-tmp-dir\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891918 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-host\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891927 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-systemd-units\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891941 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-slash\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-etc-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-device-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.891997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-os-release\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-ovnkube-config\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892067 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-device-dir\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892077 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-os-release\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-sys-fs\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.894766 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2afa112-61f1-41f2-9d75-c3647002d54b-cni-binary-copy\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-etc-openvswitch\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892208 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-sys-fs\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892254 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/654e5a71-df1b-4548-a477-fd86247ce561-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls85x\" (UniqueName: \"kubernetes.io/projected/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-kube-api-access-ls85x\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892343 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13605fab-a33b-41d6-8f26-0c2defe642af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892550 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/654e5a71-df1b-4548-a477-fd86247ce561-ovnkube-config\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.892743 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2afa112-61f1-41f2-9d75-c3647002d54b-cni-binary-copy\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.893288 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-etc-tuned\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.893616 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/654e5a71-df1b-4548-a477-fd86247ce561-ovn-node-metrics-cert\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.895710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.894291 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-tmp\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.902924 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.902887 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4wb\" (UniqueName: \"kubernetes.io/projected/8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f-kube-api-access-6m4wb\") pod \"aws-ebs-csi-driver-node-xmw6d\" (UID: \"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:57.903129 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.903104 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p42f\" (UniqueName: \"kubernetes.io/projected/13605fab-a33b-41d6-8f26-0c2defe642af-kube-api-access-4p42f\") pod \"multus-additional-cni-plugins-ttrjd\" (UID: \"13605fab-a33b-41d6-8f26-0c2defe642af\") " pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:57.903556 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.903525 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4wcx\" (UniqueName: \"kubernetes.io/projected/33c1dbe6-6285-4aba-8d84-203e5d71cbb1-kube-api-access-k4wcx\") pod \"tuned-jpt9b\" (UID: \"33c1dbe6-6285-4aba-8d84-203e5d71cbb1\") " pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:57.903773 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.903752 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlm7n\" (UniqueName: \"kubernetes.io/projected/654e5a71-df1b-4548-a477-fd86247ce561-kube-api-access-vlm7n\") pod \"ovnkube-node-bjgzw\" (UID: \"654e5a71-df1b-4548-a477-fd86247ce561\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:57.904206 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.904185 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls85x\" (UniqueName: \"kubernetes.io/projected/f9e3d2fc-85d2-4071-a95b-9ca5be60a285-kube-api-access-ls85x\") pod \"node-resolver-5kssz\" (UID: \"f9e3d2fc-85d2-4071-a95b-9ca5be60a285\") " pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:57.904346 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.904330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tm4w\" (UniqueName: \"kubernetes.io/projected/d2afa112-61f1-41f2-9d75-c3647002d54b-kube-api-access-6tm4w\") pod \"multus-kqprl\" (UID: \"d2afa112-61f1-41f2-9d75-c3647002d54b\") " pod="openshift-multus/multus-kqprl" Jul 06 23:48:57.982335 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.982301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8mxn8" Jul 06 23:48:57.989758 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.989582 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9ds8" Jul 06 23:48:57.999841 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:57.999820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:48:58.008140 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.008119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5kssz" Jul 06 23:48:58.016402 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.016383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kqprl" Jul 06 23:48:58.023696 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.023678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" Jul 06 23:48:58.032060 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.032041 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" Jul 06 23:48:58.040223 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.040206 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" Jul 06 23:48:58.049272 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.049097 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:48:58.296282 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.296201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:48:58.296282 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.296249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:58.296503 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.296370 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:48:58.296503 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.296398 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:48:58.296503 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.296424 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:48:58.296503 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.296440 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sbf9s for pod openshift-network-diagnostics/network-check-target-mptrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:48:58.296664 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.296427 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs podName:1267abdc-57d6-410e-9b07-6fa8dfdd5ac7 nodeName:}" failed. No retries permitted until 2026-07-06 23:48:59.29641161 +0000 UTC m=+3.946833539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs") pod "network-metrics-daemon-5k58z" (UID: "1267abdc-57d6-410e-9b07-6fa8dfdd5ac7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:48:58.296664 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.296548 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s podName:edba5765-8d28-4002-9d52-de6a671e6178 nodeName:}" failed. No retries permitted until 2026-07-06 23:48:59.296523818 +0000 UTC m=+3.946945748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbf9s" (UniqueName: "kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s") pod "network-check-target-mptrb" (UID: "edba5765-8d28-4002-9d52-de6a671e6178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:48:58.396975 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.396944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:58.397152 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.397074 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:48:58.397152 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.397138 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret podName:1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606 nodeName:}" failed. No retries permitted until 2026-07-06 23:48:59.397122709 +0000 UTC m=+4.047544638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret") pod "global-pull-secret-syncer-zlnnn" (UID: "1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:48:58.717177 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.717126 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:43:57 +0000 UTC" deadline="2027-12-11 02:54:37.899331546 +0000 UTC" Jul 06 23:48:58.717177 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.717174 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12531h5m39.182160331s" Jul 06 23:48:58.724100 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:58.724064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef88259_880c_4c27_a7c5_500f8b1f2698.slice/crio-0ac6aa28192d471fc444e3de08e83bdef50973ef55f94a2729745f6d0ccbdb8f WatchSource:0}: Error finding container 0ac6aa28192d471fc444e3de08e83bdef50973ef55f94a2729745f6d0ccbdb8f: Status 404 returned error can't find the container with id 0ac6aa28192d471fc444e3de08e83bdef50973ef55f94a2729745f6d0ccbdb8f Jul 06 23:48:58.725880 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:58.725749 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654e5a71_df1b_4548_a477_fd86247ce561.slice/crio-1ca7dd4c8ac282d6691a23fcac966831b90cc130e83c3d2f0e868ffbc1cb2e25 WatchSource:0}: Error finding container 1ca7dd4c8ac282d6691a23fcac966831b90cc130e83c3d2f0e868ffbc1cb2e25: Status 404 returned error can't find the container with id 1ca7dd4c8ac282d6691a23fcac966831b90cc130e83c3d2f0e868ffbc1cb2e25 Jul 06 23:48:58.729515 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:58.729494 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e3d2fc_85d2_4071_a95b_9ca5be60a285.slice/crio-4902606840d3a511da881c676a72d83c6f2f8bbfd3dfc712cc3ec02ae53cdb18 WatchSource:0}: Error finding container 4902606840d3a511da881c676a72d83c6f2f8bbfd3dfc712cc3ec02ae53cdb18: Status 404 returned error can't find the container with id 4902606840d3a511da881c676a72d83c6f2f8bbfd3dfc712cc3ec02ae53cdb18 Jul 06 23:48:58.730095 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:58.730078 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e886f2_5943_4198_afdd_3ae2ea8d90b7.slice/crio-27185b628c104ee973337062b8bc4d65272f1fdd9ffa8a437de5941370ea9e2b WatchSource:0}: Error finding container 27185b628c104ee973337062b8bc4d65272f1fdd9ffa8a437de5941370ea9e2b: Status 404 returned error can't find the container with id 27185b628c104ee973337062b8bc4d65272f1fdd9ffa8a437de5941370ea9e2b Jul 06 23:48:58.737284 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:48:58.737265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc5dbc6_451e_48ec_b38b_e19d3fb6e77f.slice/crio-e34152cc3185df1929cf2b413b2d07f4fb593f1c6c93584fb47a2d89db1782d9 WatchSource:0}: Error finding container e34152cc3185df1929cf2b413b2d07f4fb593f1c6c93584fb47a2d89db1782d9: Status 404 returned error can't find the container with id e34152cc3185df1929cf2b413b2d07f4fb593f1c6c93584fb47a2d89db1782d9 Jul 06 23:48:58.800262 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.800238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:58.800390 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:58.800366 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:48:58.807551 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.807525 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s5wl5" event={"ID":"5352efee-cf3f-463a-82ff-8e9cb779e171","Type":"ContainerStarted","Data":"a66fc8a1a73bcc0a746939962144df8ee1cb641fa1e0dbffcc084bd672415b4c"} Jul 06 23:48:58.808764 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.808744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8mxn8" event={"ID":"05e886f2-5943-4198-afdd-3ae2ea8d90b7","Type":"ContainerStarted","Data":"27185b628c104ee973337062b8bc4d65272f1fdd9ffa8a437de5941370ea9e2b"} Jul 06 23:48:58.809891 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.809870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerStarted","Data":"15eb81bd913f7888c4e49ade39d05936fdee76f0a78d8ab9270a493a2eecfcc3"} Jul 06 23:48:58.811281 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.811256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" event={"ID":"33c1dbe6-6285-4aba-8d84-203e5d71cbb1","Type":"ContainerStarted","Data":"bbd35cd687f4a36cc37f8e736c521de86531f04d0751ee5f11f9e5b9825cecca"} Jul 06 23:48:58.812384 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.812363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5kssz" event={"ID":"f9e3d2fc-85d2-4071-a95b-9ca5be60a285","Type":"ContainerStarted","Data":"4902606840d3a511da881c676a72d83c6f2f8bbfd3dfc712cc3ec02ae53cdb18"} Jul 06 23:48:58.813668 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.813645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"1ca7dd4c8ac282d6691a23fcac966831b90cc130e83c3d2f0e868ffbc1cb2e25"} Jul 06 23:48:58.814943 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.814919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9ds8" event={"ID":"1ef88259-880c-4c27-a7c5-500f8b1f2698","Type":"ContainerStarted","Data":"0ac6aa28192d471fc444e3de08e83bdef50973ef55f94a2729745f6d0ccbdb8f"} Jul 06 23:48:58.816103 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.816078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" event={"ID":"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f","Type":"ContainerStarted","Data":"e34152cc3185df1929cf2b413b2d07f4fb593f1c6c93584fb47a2d89db1782d9"} Jul 06 23:48:58.817228 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:58.817206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kqprl" event={"ID":"d2afa112-61f1-41f2-9d75-c3647002d54b","Type":"ContainerStarted","Data":"4779d57986593ba999f731a509c8d61826859b2a2fbee611d892d6984a1fdcf8"} Jul 06 23:48:59.305532 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.305446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:59.305735 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.305558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:48:59.305735 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.305717 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:48:59.305866 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.305737 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:48:59.305866 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.305752 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sbf9s for pod openshift-network-diagnostics/network-check-target-mptrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:48:59.305866 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.305813 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s podName:edba5765-8d28-4002-9d52-de6a671e6178 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:01.305791461 +0000 UTC m=+5.956213390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbf9s" (UniqueName: "kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s") pod "network-check-target-mptrb" (UID: "edba5765-8d28-4002-9d52-de6a671e6178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:48:59.306257 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.306234 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:48:59.306336 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.306297 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs podName:1267abdc-57d6-410e-9b07-6fa8dfdd5ac7 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:01.306279005 +0000 UTC m=+5.956700935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs") pod "network-metrics-daemon-5k58z" (UID: "1267abdc-57d6-410e-9b07-6fa8dfdd5ac7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:48:59.406216 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.406183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:48:59.406403 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.406357 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:48:59.406470 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.406414 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret podName:1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:01.406396231 +0000 UTC m=+6.056818170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret") pod "global-pull-secret-syncer-zlnnn" (UID: "1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:48:59.718182 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.718083 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:43:57 +0000 UTC" deadline="2028-02-24 17:57:51.965016578 +0000 UTC" Jul 06 23:48:59.718182 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.718129 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14346h8m52.246892754s" Jul 06 23:48:59.801687 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.801654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:48:59.801687 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.801679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:48:59.801910 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.801787 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:48:59.802234 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:48:59.802166 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:48:59.833287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.833245 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f8ecbc7fda6a812262efefe3033234e" containerID="acf4e0aa0bbbb6e73c7153444a2cb19181d98453ad695852dba059b1f3eb6969" exitCode=0 Jul 06 23:48:59.838921 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.833985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"7f8ecbc7fda6a812262efefe3033234e","Type":"ContainerDied","Data":"acf4e0aa0bbbb6e73c7153444a2cb19181d98453ad695852dba059b1f3eb6969"} Jul 06 23:48:59.838921 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:48:59.838229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"319f82ace8919ccdff136f05eb05d345","Type":"ContainerStarted","Data":"ee20baf99359607190276ea665cecc4da4a22cfeecc73364255cd730291f49c3"} Jul 06 23:49:00.799685 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:00.799649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:00.800166 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:00.799781 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:00.856665 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:00.856632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"7f8ecbc7fda6a812262efefe3033234e","Type":"ContainerStarted","Data":"079855df799001a670d5576acb7c5e830822dd96e4b55869003477bfdd7dc6d4"} Jul 06 23:49:00.871112 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:00.871058 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" podStartSLOduration=3.871038483 podStartE2EDuration="3.871038483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:48:59.868312741 +0000 UTC m=+4.518734692" watchObservedRunningTime="2026-07-06 23:49:00.871038483 +0000 UTC m=+5.521460430" Jul 06 23:49:01.324575 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:01.324521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:01.324762 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:01.324623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:01.325314 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.324875 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:01.325314 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.324947 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs podName:1267abdc-57d6-410e-9b07-6fa8dfdd5ac7 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:05.32492501 +0000 UTC m=+9.975346937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs") pod "network-metrics-daemon-5k58z" (UID: "1267abdc-57d6-410e-9b07-6fa8dfdd5ac7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:01.325314 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.325000 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:49:01.325314 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.325029 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:49:01.325314 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.325043 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sbf9s for pod openshift-network-diagnostics/network-check-target-mptrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:01.325314 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.325123 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s podName:edba5765-8d28-4002-9d52-de6a671e6178 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:05.325106597 +0000 UTC m=+9.975528528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbf9s" (UniqueName: "kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s") pod "network-check-target-mptrb" (UID: "edba5765-8d28-4002-9d52-de6a671e6178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:01.425912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:01.425873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:01.426069 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.426019 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:01.426139 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.426082 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret podName:1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:05.426063873 +0000 UTC m=+10.076485820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret") pod "global-pull-secret-syncer-zlnnn" (UID: "1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:01.803053 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:01.802281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:01.803053 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.802422 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:01.803053 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:01.802821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:01.803053 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:01.802904 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:02.799449 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:02.799411 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:02.799636 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:02.799564 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:03.799731 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:03.799700 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:03.800109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:03.799710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:03.800109 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:03.799830 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:03.800109 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:03.799934 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:04.800252 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:04.800211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:04.800755 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:04.800353 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:05.360297 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:05.360263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:05.360521 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:05.360347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:05.360521 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.360472 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:49:05.360521 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.360490 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:49:05.360521 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.360520 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sbf9s for pod openshift-network-diagnostics/network-check-target-mptrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:05.360724 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.360576 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s podName:edba5765-8d28-4002-9d52-de6a671e6178 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:13.360558293 +0000 UTC m=+18.010980225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbf9s" (UniqueName: "kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s") pod "network-check-target-mptrb" (UID: "edba5765-8d28-4002-9d52-de6a671e6178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:05.360724 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.360598 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:05.360724 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.360636 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs podName:1267abdc-57d6-410e-9b07-6fa8dfdd5ac7 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:13.360625372 +0000 UTC m=+18.011047301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs") pod "network-metrics-daemon-5k58z" (UID: "1267abdc-57d6-410e-9b07-6fa8dfdd5ac7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:05.461108 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:05.461070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:05.461284 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.461214 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:05.461284 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.461263 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret podName:1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:13.461249241 +0000 UTC m=+18.111671165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret") pod "global-pull-secret-syncer-zlnnn" (UID: "1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:05.800445 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:05.800374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:05.800893 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.800486 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:05.800893 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:05.800877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:05.801002 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:05.800979 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:06.799372 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:06.799342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:06.799560 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:06.799468 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:07.801010 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:07.800978 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:07.801445 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:07.801125 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:07.801960 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:07.801824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:07.801960 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:07.801922 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:08.799685 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:08.799653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:08.799845 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:08.799795 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:09.799685 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:09.799615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:09.800091 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:09.799746 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:09.800091 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:09.799796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:09.800091 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:09.799906 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:10.799206 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:10.799180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:10.799396 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:10.799294 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:11.799712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:11.799677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:11.800153 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:11.799819 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:11.800153 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:11.799906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:11.800153 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:11.800018 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:12.800207 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:12.800173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:12.800713 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:12.800295 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:13.423667 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:13.423636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:13.423815 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:13.423686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:13.423858 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.423822 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:49:13.423858 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.423843 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:13.423858 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.423846 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:49:13.423945 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.423861 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sbf9s for pod openshift-network-diagnostics/network-check-target-mptrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:13.423945 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.423902 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs podName:1267abdc-57d6-410e-9b07-6fa8dfdd5ac7 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:29.423884933 +0000 UTC m=+34.074306858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs") pod "network-metrics-daemon-5k58z" (UID: "1267abdc-57d6-410e-9b07-6fa8dfdd5ac7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:13.423945 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.423916 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s podName:edba5765-8d28-4002-9d52-de6a671e6178 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:29.423910028 +0000 UTC m=+34.074331952 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbf9s" (UniqueName: "kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s") pod "network-check-target-mptrb" (UID: "edba5765-8d28-4002-9d52-de6a671e6178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:13.524039 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:13.523970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:13.524182 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.524112 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:13.524182 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.524176 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret podName:1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:29.524160283 +0000 UTC m=+34.174582208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret") pod "global-pull-secret-syncer-zlnnn" (UID: "1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:13.799243 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:13.799167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:13.799398 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.799302 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:13.799398 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:13.799344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:13.799528 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:13.799453 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:14.799762 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:14.799730 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:14.800151 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:14.799875 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:15.800662 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:15.800632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:15.801071 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:15.800728 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:15.801071 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:15.800813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:15.801071 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:15.800933 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:16.799787 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:16.799749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:16.799975 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:16.799878 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:17.802295 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.801664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:17.802295 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:17.801782 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:17.802295 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.802151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:17.802295 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:17.802249 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:17.892395 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.891674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s5wl5" event={"ID":"5352efee-cf3f-463a-82ff-8e9cb779e171","Type":"ContainerStarted","Data":"d2321cdb643bbcd701022f2d6f62a50fca724f0f6bef30ff8138c30257dbce6e"} Jul 06 23:49:17.896374 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.894828 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerStarted","Data":"1bb6f325e2fd7cc96900510ddcf8330a74c85e6c96377926c06f8c9faea48ef1"} Jul 06 23:49:17.900105 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.899329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" event={"ID":"33c1dbe6-6285-4aba-8d84-203e5d71cbb1","Type":"ContainerStarted","Data":"afe36ee646306b0ca852f413a98090e5dc03a1bfecf432cfa9ae9d92cdb40839"} Jul 06 23:49:17.905695 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.905566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"d23d8bb9952180a13bda6a2268281f17d2279187bdb6d16d3861f9b693ca4267"} Jul 06 23:49:17.910464 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.910293 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s5wl5" podStartSLOduration=7.397690015 podStartE2EDuration="21.910273314s" totalImagesPullingTime="14.512583299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:17.909344268 +0000 UTC m=+22.559766241" watchObservedRunningTime="2026-07-06 23:49:17.910273314 +0000 UTC m=+22.560695318" Jul 06 23:49:17.912153 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.912101 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" podStartSLOduration=20.912078163 podStartE2EDuration="20.912078163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:00.87199676 +0000 UTC m=+5.522418716" watchObservedRunningTime="2026-07-06 23:49:17.912078163 +0000 UTC m=+22.562500110" Jul 06 23:49:17.920284 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.920090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kqprl" event={"ID":"d2afa112-61f1-41f2-9d75-c3647002d54b","Type":"ContainerStarted","Data":"b60f6b6d01ca180c3c7458d47deab00ccca014020938ffd0dc53b0c680cf7d1c"} Jul 06 23:49:17.933794 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.933569 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jpt9b" podStartSLOduration=2.9844816339999998 podStartE2EDuration="21.933550662s" totalImagesPullingTime="18.949069028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:17.932570281 +0000 UTC m=+22.582992248" watchObservedRunningTime="2026-07-06 23:49:17.933550662 +0000 UTC m=+22.583972610" Jul 06 23:49:17.972051 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:17.972002 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s9ds8" podStartSLOduration=3.017571776 podStartE2EDuration="21.971982609s" totalImagesPullingTime="18.954410833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:17.947437109 +0000 UTC m=+22.597859055" watchObservedRunningTime="2026-07-06 23:49:17.971982609 +0000 UTC m=+22.622404557" Jul 06 23:49:18.799167 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.799141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:18.799300 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:18.799239 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:18.845978 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.845954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:49:18.846474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.846459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:49:18.861200 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.861163 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kqprl" podStartSLOduration=3.861603553 podStartE2EDuration="22.861149853s" totalImagesPullingTime="18.9995463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:17.987628526 +0000 UTC m=+22.638050512" watchObservedRunningTime="2026-07-06 23:49:18.861149853 +0000 UTC m=+23.511571800" Jul 06 23:49:18.923197 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.923177 2564 generic.go:356] "Generic (PLEG): container finished" podID="13605fab-a33b-41d6-8f26-0c2defe642af" containerID="1bb6f325e2fd7cc96900510ddcf8330a74c85e6c96377926c06f8c9faea48ef1" exitCode=0 Jul 06 23:49:18.923315 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.923239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerDied","Data":"1bb6f325e2fd7cc96900510ddcf8330a74c85e6c96377926c06f8c9faea48ef1"} Jul 06 23:49:18.924701 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.924674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5kssz" event={"ID":"f9e3d2fc-85d2-4071-a95b-9ca5be60a285","Type":"ContainerStarted","Data":"3c30d19c4e9f3ac7cb49ba2c71df12387ee1c570214b6a6d5c13e3e7930cb454"} Jul 06 23:49:18.927478 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.927460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"b55deb967da86c3748aa8a3c29b9655373d90a416bc6da661467d667ad96060d"} Jul 06 23:49:18.927584 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.927539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"9933754df3faef1776f255a53e2949d2977d981471098f2b6c4ca732df6a6666"} Jul 06 23:49:18.927630 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.927587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"890ab271c5ef36651fd8571a9a2971292aa2c1b1c2325acd998ef1631720a5f3"} Jul 06 23:49:18.927630 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.927617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"9b45d5eceb34ac0b4424eb60b7eaeecbd141228379d37022bccd14efd7c7362d"} Jul 06 23:49:18.927717 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.927643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"03f9549d7db6e0d466316ea74555d0cd39b9e1d87541ed5d6721e185a0edc8e7"} Jul 06 23:49:18.928758 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.928738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9ds8" event={"ID":"1ef88259-880c-4c27-a7c5-500f8b1f2698","Type":"ContainerStarted","Data":"bf915864e615dc77f2d578ed6c177d5b3f4b6c40f100b79483a0598e80fd2b34"} Jul 06 23:49:18.930091 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.930068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" event={"ID":"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f","Type":"ContainerStarted","Data":"805ac8d5521983327dd6a0a46a3f40cabcdfbedc5e232450b1ad330815e437bf"} Jul 06 23:49:18.930548 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.930528 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:49:18.930951 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.930938 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s5wl5" Jul 06 23:49:18.970817 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:18.970750 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5kssz" podStartSLOduration=4.023332027 podStartE2EDuration="22.970738203s" totalImagesPullingTime="18.947406176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:18.970404531 +0000 UTC m=+23.620826478" watchObservedRunningTime="2026-07-06 23:49:18.970738203 +0000 UTC m=+23.621160150" Jul 06 23:49:19.161436 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.161402 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 23:49:19.734913 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.734813 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T23:49:19.161427436Z","UUID":"cf651ebb-256e-4ae0-8987-9209346e52fb","Handler":null,"Name":"","Endpoint":""} Jul 06 23:49:19.736513 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.736462 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 23:49:19.736644 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.736526 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 23:49:19.799579 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.799545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:19.799729 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:19.799657 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:19.799729 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.799720 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:19.799883 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:19.799817 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:19.934639 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.934601 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8mxn8" event={"ID":"05e886f2-5943-4198-afdd-3ae2ea8d90b7","Type":"ContainerStarted","Data":"1b2339596591189c5730b81c66fd6b58acdc4b2d3b60698bdd7053011ea1ac86"} Jul 06 23:49:19.937715 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.937560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" event={"ID":"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f","Type":"ContainerStarted","Data":"e336eec433fe9f163616a9fb16aa75b5d7e00aaf44544c4e1fcce26568a04bd7"} Jul 06 23:49:19.949879 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:19.949835 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8mxn8" podStartSLOduration=5.002222477 podStartE2EDuration="23.949820231s" totalImagesPullingTime="18.947597754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:19.94920907 +0000 UTC m=+24.599631017" watchObservedRunningTime="2026-07-06 23:49:19.949820231 +0000 UTC m=+24.600242178" Jul 06 23:49:20.800227 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:20.800149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:20.800473 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:20.800278 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:20.943905 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:20.943867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"36f78de100768c94c10cdbc8c5a9f855d3cc662a52bdf1b09042faa7ecbb21b9"} Jul 06 23:49:20.946351 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:20.946308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" event={"ID":"8cc5dbc6-451e-48ec-b38b-e19d3fb6e77f","Type":"ContainerStarted","Data":"930da205ca79f47733e7e5327478426d09f9c66009ecc86823ceae9c461abdf9"} Jul 06 23:49:20.963360 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:20.963318 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmw6d" podStartSLOduration=3.908311068 podStartE2EDuration="24.963298867s" totalImagesPullingTime="21.054987799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:20.962751055 +0000 UTC m=+25.613173006" watchObservedRunningTime="2026-07-06 23:49:20.963298867 +0000 UTC m=+25.613720816" Jul 06 23:49:21.799924 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:21.799891 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:21.800094 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:21.800010 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:21.800094 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:21.800078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:21.800217 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:21.800193 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:22.799982 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:22.799957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:22.800577 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:22.800057 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:23.800182 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:23.800019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:23.800680 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:23.800028 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:23.800680 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:23.800254 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:23.800680 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:23.800362 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:23.954953 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:23.954921 2564 generic.go:356] "Generic (PLEG): container finished" podID="13605fab-a33b-41d6-8f26-0c2defe642af" containerID="7ef93730b02d5abd29546c63a5ed0f75cf902c0334d19e17620d67b0b13d4c09" exitCode=0 Jul 06 23:49:23.955104 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:23.954976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerDied","Data":"7ef93730b02d5abd29546c63a5ed0f75cf902c0334d19e17620d67b0b13d4c09"} Jul 06 23:49:23.958805 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:23.958774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" event={"ID":"654e5a71-df1b-4548-a477-fd86247ce561","Type":"ContainerStarted","Data":"73125f8d7dff5a8e4dbf66fe92ab33d87b292eff5493d98fb9287f22b80bb55e"} Jul 06 23:49:23.959108 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:23.959088 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:49:23.975054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:23.975034 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:49:24.015967 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:24.015931 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" podStartSLOduration=8.990353542 podStartE2EDuration="28.015918624s" totalImagesPullingTime="19.025565082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:24.015545464 +0000 UTC m=+28.665967420" watchObservedRunningTime="2026-07-06 23:49:24.015918624 +0000 UTC m=+28.666340571" Jul 06 23:49:24.799677 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:24.799489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:24.799940 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:24.799748 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:24.961062 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:24.961030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:49:24.961458 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:24.961113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:49:24.975841 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:24.975821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:49:25.172180 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.172157 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5k58z"] Jul 06 23:49:25.172317 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.172262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:25.172393 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:25.172356 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:25.172801 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.172782 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlnnn"] Jul 06 23:49:25.172878 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.172856 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:25.172949 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:25.172933 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:25.182793 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.182763 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mptrb"] Jul 06 23:49:25.182903 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.182869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:25.182966 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:25.182947 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:25.964280 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.964246 2564 generic.go:356] "Generic (PLEG): container finished" podID="13605fab-a33b-41d6-8f26-0c2defe642af" containerID="7e9ed039219558f05336835a56f1b7ac537c52f6e2b4f10ab47f6875aaebfa74" exitCode=0 Jul 06 23:49:25.964989 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:25.964335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerDied","Data":"7e9ed039219558f05336835a56f1b7ac537c52f6e2b4f10ab47f6875aaebfa74"} Jul 06 23:49:26.799633 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:26.799597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:26.799760 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:26.799703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:26.799760 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:26.799709 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:26.799760 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:26.799732 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:26.799855 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:26.799815 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:26.799898 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:26.799883 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:27.970680 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:27.970634 2564 generic.go:356] "Generic (PLEG): container finished" podID="13605fab-a33b-41d6-8f26-0c2defe642af" containerID="ea4ed9ae56b76079af0ea0687efa2a5d72f5213829e4999f9d23d68fbe3faaf3" exitCode=0 Jul 06 23:49:27.971141 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:27.970695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerDied","Data":"ea4ed9ae56b76079af0ea0687efa2a5d72f5213829e4999f9d23d68fbe3faaf3"} Jul 06 23:49:28.800149 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:28.800061 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:28.800149 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:28.800102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:28.800341 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:28.800067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:28.800341 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:28.800184 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5k58z" podUID="1267abdc-57d6-410e-9b07-6fa8dfdd5ac7" Jul 06 23:49:28.800341 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:28.800255 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mptrb" podUID="edba5765-8d28-4002-9d52-de6a671e6178" Jul 06 23:49:28.800341 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:28.800332 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlnnn" podUID="1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606" Jul 06 23:49:29.445428 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:29.445216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:29.445817 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.445428 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:49:29.445817 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.445457 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:49:29.445817 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:29.445466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:29.445817 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.445470 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sbf9s for pod openshift-network-diagnostics/network-check-target-mptrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:29.445817 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.445551 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:29.445817 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.445609 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s podName:edba5765-8d28-4002-9d52-de6a671e6178 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:01.445590158 +0000 UTC m=+66.096012085 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbf9s" (UniqueName: "kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s") pod "network-check-target-mptrb" (UID: "edba5765-8d28-4002-9d52-de6a671e6178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:49:29.445817 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.445626 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs podName:1267abdc-57d6-410e-9b07-6fa8dfdd5ac7 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:01.445618886 +0000 UTC m=+66.096040812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs") pod "network-metrics-daemon-5k58z" (UID: "1267abdc-57d6-410e-9b07-6fa8dfdd5ac7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:49:29.546332 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:29.546304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:29.546520 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.546408 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:29.546520 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:29.546464 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret podName:1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:01.546451633 +0000 UTC m=+66.196873558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret") pod "global-pull-secret-syncer-zlnnn" (UID: "1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:49:30.705815 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.705790 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeReady" Jul 06 23:49:30.706256 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.705933 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 23:49:30.749355 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.749324 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jn4x7"] Jul 06 23:49:30.754853 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.754831 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7kbvg"] Jul 06 23:49:30.754995 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.754946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.757282 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.757258 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 23:49:30.757396 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.757258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p99x7\"" Jul 06 23:49:30.757396 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.757293 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 23:49:30.758853 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.758833 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:30.761042 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.760770 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 23:49:30.761042 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.760849 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 23:49:30.761042 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.760866 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hk7zr\"" Jul 06 23:49:30.761042 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.760866 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 23:49:30.765904 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.765883 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7kbvg"] Jul 06 23:49:30.766635 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.766616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jn4x7"] Jul 06 23:49:30.800124 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.800098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:49:30.800124 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.800118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:49:30.800124 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.800129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:49:30.818654 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.818623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:49:30.818771 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.818671 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:49:30.818771 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.818737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:49:30.819015 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.819001 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m58d\"" Jul 06 23:49:30.819251 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.819234 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7lfhk\"" Jul 06 23:49:30.819340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.819275 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:49:30.856988 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.856962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgbm\" (UniqueName: \"kubernetes.io/projected/9cf530db-a95a-469f-9bde-41d882c205ab-kube-api-access-4jgbm\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.856988 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.856991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4529a4c7-0153-476f-bf94-566619bd5d22-cert\") pod \"ingress-canary-7kbvg\" (UID: \"4529a4c7-0153-476f-bf94-566619bd5d22\") " pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:30.857180 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.857012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf530db-a95a-469f-9bde-41d882c205ab-tmp-dir\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.857180 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.857029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflfs\" (UniqueName: \"kubernetes.io/projected/4529a4c7-0153-476f-bf94-566619bd5d22-kube-api-access-wflfs\") pod \"ingress-canary-7kbvg\" (UID: \"4529a4c7-0153-476f-bf94-566619bd5d22\") " pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:30.857180 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.857124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.857329 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.857181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf530db-a95a-469f-9bde-41d882c205ab-config-volume\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.957754 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.957654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflfs\" (UniqueName: \"kubernetes.io/projected/4529a4c7-0153-476f-bf94-566619bd5d22-kube-api-access-wflfs\") pod \"ingress-canary-7kbvg\" (UID: \"4529a4c7-0153-476f-bf94-566619bd5d22\") " pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:30.957754 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.957725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.957927 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.957782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf530db-a95a-469f-9bde-41d882c205ab-config-volume\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.957927 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:30.957898 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:49:30.958018 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:30.957969 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls podName:9cf530db-a95a-469f-9bde-41d882c205ab nodeName:}" failed. No retries permitted until 2026-07-06 23:49:31.457946555 +0000 UTC m=+36.108368498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls") pod "dns-default-jn4x7" (UID: "9cf530db-a95a-469f-9bde-41d882c205ab") : secret "dns-default-metrics-tls" not found Jul 06 23:49:30.958087 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.958045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgbm\" (UniqueName: \"kubernetes.io/projected/9cf530db-a95a-469f-9bde-41d882c205ab-kube-api-access-4jgbm\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.958087 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.958078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4529a4c7-0153-476f-bf94-566619bd5d22-cert\") pod \"ingress-canary-7kbvg\" (UID: \"4529a4c7-0153-476f-bf94-566619bd5d22\") " pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:30.958174 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.958107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf530db-a95a-469f-9bde-41d882c205ab-tmp-dir\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.958386 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.958362 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf530db-a95a-469f-9bde-41d882c205ab-config-volume\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.958386 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.958378 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf530db-a95a-469f-9bde-41d882c205ab-tmp-dir\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.963083 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.963062 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4529a4c7-0153-476f-bf94-566619bd5d22-cert\") pod \"ingress-canary-7kbvg\" (UID: \"4529a4c7-0153-476f-bf94-566619bd5d22\") " pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:30.980082 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.980056 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgbm\" (UniqueName: \"kubernetes.io/projected/9cf530db-a95a-469f-9bde-41d882c205ab-kube-api-access-4jgbm\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:30.981109 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:30.981087 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflfs\" (UniqueName: \"kubernetes.io/projected/4529a4c7-0153-476f-bf94-566619bd5d22-kube-api-access-wflfs\") pod \"ingress-canary-7kbvg\" (UID: \"4529a4c7-0153-476f-bf94-566619bd5d22\") " pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:31.074139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:31.074100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7kbvg" Jul 06 23:49:31.461904 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:31.461866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:31.462073 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:31.462019 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:49:31.462136 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:31.462085 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls podName:9cf530db-a95a-469f-9bde-41d882c205ab nodeName:}" failed. No retries permitted until 2026-07-06 23:49:32.462067708 +0000 UTC m=+37.112489634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls") pod "dns-default-jn4x7" (UID: "9cf530db-a95a-469f-9bde-41d882c205ab") : secret "dns-default-metrics-tls" not found Jul 06 23:49:32.038320 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:32.038296 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7kbvg"] Jul 06 23:49:32.042131 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:49:32.042107 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4529a4c7_0153_476f_bf94_566619bd5d22.slice/crio-00a9dce784f634992e2f65b40f563a7790a72ca26eee5ae25e944135a5534e40 WatchSource:0}: Error finding container 00a9dce784f634992e2f65b40f563a7790a72ca26eee5ae25e944135a5534e40: Status 404 returned error can't find the container with id 00a9dce784f634992e2f65b40f563a7790a72ca26eee5ae25e944135a5534e40 Jul 06 23:49:32.468454 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:32.468426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:32.468636 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:32.468607 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:49:32.468701 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:32.468674 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls podName:9cf530db-a95a-469f-9bde-41d882c205ab nodeName:}" failed. No retries permitted until 2026-07-06 23:49:34.468658473 +0000 UTC m=+39.119080398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls") pod "dns-default-jn4x7" (UID: "9cf530db-a95a-469f-9bde-41d882c205ab") : secret "dns-default-metrics-tls" not found Jul 06 23:49:32.984212 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:32.984171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7kbvg" event={"ID":"4529a4c7-0153-476f-bf94-566619bd5d22","Type":"ContainerStarted","Data":"00a9dce784f634992e2f65b40f563a7790a72ca26eee5ae25e944135a5534e40"} Jul 06 23:49:32.987199 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:32.987174 2564 generic.go:356] "Generic (PLEG): container finished" podID="13605fab-a33b-41d6-8f26-0c2defe642af" containerID="b2200cacc7c045e8fa4552332825333e26de4bbb6fce13342aae68ee274f8eeb" exitCode=0 Jul 06 23:49:32.987310 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:32.987228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerDied","Data":"b2200cacc7c045e8fa4552332825333e26de4bbb6fce13342aae68ee274f8eeb"} Jul 06 23:49:33.992765 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:33.992733 2564 generic.go:356] "Generic (PLEG): container finished" podID="13605fab-a33b-41d6-8f26-0c2defe642af" containerID="c3cd413d0ba06935dfa56bb428fd8783493881873dc01f7c44efa1fa5647699a" exitCode=0 Jul 06 23:49:33.993098 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:33.992798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerDied","Data":"c3cd413d0ba06935dfa56bb428fd8783493881873dc01f7c44efa1fa5647699a"} Jul 06 23:49:34.485252 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:34.485227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:34.485377 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:34.485361 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:49:34.485464 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:34.485454 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls podName:9cf530db-a95a-469f-9bde-41d882c205ab nodeName:}" failed. No retries permitted until 2026-07-06 23:49:38.485437913 +0000 UTC m=+43.135859837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls") pod "dns-default-jn4x7" (UID: "9cf530db-a95a-469f-9bde-41d882c205ab") : secret "dns-default-metrics-tls" not found Jul 06 23:49:34.996808 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:34.996775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7kbvg" event={"ID":"4529a4c7-0153-476f-bf94-566619bd5d22","Type":"ContainerStarted","Data":"070334e0128b549ba7c249d202bd057239175176c67cae9a1764f562706f6a3f"} Jul 06 23:49:35.000157 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:35.000128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" event={"ID":"13605fab-a33b-41d6-8f26-0c2defe642af","Type":"ContainerStarted","Data":"b729d4ecf36be8aea2ecef56699acf9061793aed2ed11e56ee531a29519bdbdc"} Jul 06 23:49:35.013011 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:35.012957 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7kbvg" podStartSLOduration=2.593335948 podStartE2EDuration="5.012944169s" totalImagesPullingTime="2.419608221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:35.012273297 +0000 UTC m=+39.662695265" watchObservedRunningTime="2026-07-06 23:49:35.012944169 +0000 UTC m=+39.663366115" Jul 06 23:49:36.608471 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.608421 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ttrjd" podStartSLOduration=11.536510825 podStartE2EDuration="40.608404202s" totalImagesPullingTime="29.071893377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:35.038553167 +0000 UTC m=+39.688975134" watchObservedRunningTime="2026-07-06 23:49:36.608404202 +0000 UTC m=+41.258826149" Jul 06 23:49:36.608854 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.608749 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8"] Jul 06 23:49:36.642909 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.642887 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xgj9q"] Jul 06 23:49:36.643035 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.643004 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" Jul 06 23:49:36.646707 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.646687 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.646834 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.646717 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jrs9l\"" Jul 06 23:49:36.649824 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.649809 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.674352 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.672675 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-65b4d79d85-rmpz9"] Jul 06 23:49:36.674548 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.674398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.676983 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.676954 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 23:49:36.677101 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.677065 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.678013 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.677994 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.682033 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.682015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8"] Jul 06 23:49:36.682117 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.682041 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xgj9q"] Jul 06 23:49:36.682117 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.682105 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.684700 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.684681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-44gnv\"" Jul 06 23:49:36.684841 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.684720 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 23:49:36.685515 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.685471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65b4d79d85-rmpz9"] Jul 06 23:49:36.686323 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.686305 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 23:49:36.686417 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.686337 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 23:49:36.686485 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.686477 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.689884 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.689869 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.690723 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.690711 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 23:49:36.690838 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.690823 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 23:49:36.691013 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.691003 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-njfrp\"" Jul 06 23:49:36.694974 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.694959 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 23:49:36.705089 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bdd76a4e-600e-458e-adb3-801d5f30c4be-snapshots\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.705169 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd76a4e-600e-458e-adb3-801d5f30c4be-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.705169 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8m8\" (UniqueName: \"kubernetes.io/projected/beb31d50-ff75-48a0-afc9-42ce85ddf86d-kube-api-access-jh8m8\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.705169 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxk6\" (UniqueName: \"kubernetes.io/projected/bdd76a4e-600e-458e-adb3-801d5f30c4be-kube-api-access-2fxk6\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.705287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.705287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6w94\" (UniqueName: \"kubernetes.io/projected/49c1d9cf-b5e4-47b5-ae63-77ea95565764-kube-api-access-g6w94\") pod \"volume-data-source-validator-5d456fdbcd-c9nx8\" (UID: \"49c1d9cf-b5e4-47b5-ae63-77ea95565764\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" Jul 06 23:49:36.705287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd76a4e-600e-458e-adb3-801d5f30c4be-tmp\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.705287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-default-certificate\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.705287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bdd76a4e-600e-458e-adb3-801d5f30c4be-serving-cert\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.705424 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-stats-auth\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.705424 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd76a4e-600e-458e-adb3-801d5f30c4be-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.705424 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.705349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.713605 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.713583 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s"] Jul 06 23:49:36.734040 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.734022 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z"] Jul 06 23:49:36.734163 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.734146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.735946 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.735926 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-v46sf\"" Jul 06 23:49:36.736149 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.736134 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.736149 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.736140 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 23:49:36.736488 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.736470 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 23:49:36.736607 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.736471 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.752045 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.752022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.752115 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.752027 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lz94n"] Jul 06 23:49:36.753987 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.753972 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 23:49:36.754774 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.754759 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-g5nvr\"" Jul 06 23:49:36.754878 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.754783 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 23:49:36.754878 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.754825 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.755080 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.755055 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.766026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.766007 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s"] Jul 06 23:49:36.766103 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.766038 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z"] Jul 06 23:49:36.766103 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.766052 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lz94n"] Jul 06 23:49:36.766170 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.766103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.767938 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.767915 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.768023 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.767999 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 23:49:36.768133 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.768119 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.768203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.768189 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 23:49:36.768574 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.768557 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gkzxx\"" Jul 06 23:49:36.774426 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.774409 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 23:49:36.806468 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.806581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-config\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.806581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3acadf3-310e-4f27-99ed-c62ee39567a1-config\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.806581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.806695 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxk6\" (UniqueName: \"kubernetes.io/projected/bdd76a4e-600e-458e-adb3-801d5f30c4be-kube-api-access-2fxk6\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.806695 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd76a4e-600e-458e-adb3-801d5f30c4be-tmp\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.806695 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bdd76a4e-600e-458e-adb3-801d5f30c4be-serving-cert\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.806804 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsml6\" (UniqueName: \"kubernetes.io/projected/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-kube-api-access-dsml6\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.806804 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zn4h\" (UniqueName: \"kubernetes.io/projected/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-kube-api-access-5zn4h\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.806804 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.806950 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bdd76a4e-600e-458e-adb3-801d5f30c4be-snapshots\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.806950 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd76a4e-600e-458e-adb3-801d5f30c4be-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.806950 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3acadf3-310e-4f27-99ed-c62ee39567a1-serving-cert\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.806950 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:36.806928 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:37.306912596 +0000 UTC m=+41.957334521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : configmap references non-existent config key: service-ca.crt Jul 06 23:49:36.807147 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3acadf3-310e-4f27-99ed-c62ee39567a1-trusted-ca\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.807147 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.806996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6w94\" (UniqueName: \"kubernetes.io/projected/49c1d9cf-b5e4-47b5-ae63-77ea95565764-kube-api-access-g6w94\") pod \"volume-data-source-validator-5d456fdbcd-c9nx8\" (UID: \"49c1d9cf-b5e4-47b5-ae63-77ea95565764\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" Jul 06 23:49:36.807147 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8m8\" (UniqueName: \"kubernetes.io/projected/beb31d50-ff75-48a0-afc9-42ce85ddf86d-kube-api-access-jh8m8\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.807147 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7kdf\" (UniqueName: \"kubernetes.io/projected/a3acadf3-310e-4f27-99ed-c62ee39567a1-kube-api-access-g7kdf\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.807147 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.807147 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-default-certificate\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.807147 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807111 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdd76a4e-600e-458e-adb3-801d5f30c4be-tmp\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.807479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd76a4e-600e-458e-adb3-801d5f30c4be-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.807479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807315 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bdd76a4e-600e-458e-adb3-801d5f30c4be-snapshots\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.807479 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:36.807334 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:49:36.807479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807372 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.807479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807373 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd76a4e-600e-458e-adb3-801d5f30c4be-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.807479 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:36.807382 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:37.30736498 +0000 UTC m=+41.957786918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : secret "router-metrics-certs-default" not found Jul 06 23:49:36.807479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.807425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-stats-auth\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.808035 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.808014 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd76a4e-600e-458e-adb3-801d5f30c4be-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.810254 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.810227 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bdd76a4e-600e-458e-adb3-801d5f30c4be-serving-cert\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.810363 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.810348 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-default-certificate\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.810518 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.810485 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-stats-auth\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.825423 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.825399 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb"] Jul 06 23:49:36.826675 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.826656 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxk6\" (UniqueName: \"kubernetes.io/projected/bdd76a4e-600e-458e-adb3-801d5f30c4be-kube-api-access-2fxk6\") pod \"insights-operator-5c847dcfc4-xgj9q\" (UID: \"bdd76a4e-600e-458e-adb3-801d5f30c4be\") " pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:36.827637 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.827621 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8m8\" (UniqueName: \"kubernetes.io/projected/beb31d50-ff75-48a0-afc9-42ce85ddf86d-kube-api-access-jh8m8\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:36.828621 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.828598 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6w94\" (UniqueName: \"kubernetes.io/projected/49c1d9cf-b5e4-47b5-ae63-77ea95565764-kube-api-access-g6w94\") pod \"volume-data-source-validator-5d456fdbcd-c9nx8\" (UID: \"49c1d9cf-b5e4-47b5-ae63-77ea95565764\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" Jul 06 23:49:36.862583 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.862520 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4"] Jul 06 23:49:36.862583 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.862576 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" Jul 06 23:49:36.864894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.864874 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-l5m7m\"" Jul 06 23:49:36.878577 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.878559 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf"] Jul 06 23:49:36.878682 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.878667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:36.880599 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.880580 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hd56n\"" Jul 06 23:49:36.880701 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.880684 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.881036 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.881022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 23:49:36.881126 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.881113 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.892473 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.892457 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f7d5b7dd4-88rkl"] Jul 06 23:49:36.892583 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.892559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:36.894764 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.894745 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 23:49:36.894920 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.894904 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 23:49:36.895565 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.895542 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 23:49:36.895630 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.895571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-np7xr\"" Jul 06 23:49:36.895869 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.895855 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 23:49:36.908035 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.908160 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sngzr\" (UniqueName: \"kubernetes.io/projected/7b6d7f24-f100-4629-8acc-2c15f99632bd-kube-api-access-sngzr\") pod \"network-check-source-989f8b8dc-tqzsb\" (UID: \"7b6d7f24-f100-4629-8acc-2c15f99632bd\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" Jul 06 23:49:36.908160 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.908160 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-config\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.908160 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3acadf3-310e-4f27-99ed-c62ee39567a1-config\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.908725 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908704 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb"] Jul 06 23:49:36.908843 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908740 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4"] Jul 06 23:49:36.908843 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf"] Jul 06 23:49:36.908843 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f7d5b7dd4-88rkl"] Jul 06 23:49:36.908843 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908814 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908852 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908906 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3acadf3-310e-4f27-99ed-c62ee39567a1-config\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62f2w\" (UniqueName: \"kubernetes.io/projected/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-kube-api-access-62f2w\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.908964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsml6\" (UniqueName: \"kubernetes.io/projected/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-kube-api-access-dsml6\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zn4h\" (UniqueName: \"kubernetes.io/projected/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-kube-api-access-5zn4h\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909063 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-config\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3acadf3-310e-4f27-99ed-c62ee39567a1-serving-cert\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3acadf3-310e-4f27-99ed-c62ee39567a1-trusted-ca\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.909275 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7kdf\" (UniqueName: \"kubernetes.io/projected/a3acadf3-310e-4f27-99ed-c62ee39567a1-kube-api-access-g7kdf\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.909886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nldz\" (UniqueName: \"kubernetes.io/projected/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-kube-api-access-2nldz\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:36.909886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.909324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:36.910443 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.910419 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3acadf3-310e-4f27-99ed-c62ee39567a1-trusted-ca\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.910918 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.910902 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 23:49:36.911038 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.911018 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.911150 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.911133 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 23:49:36.911599 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.911582 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5vhf7\"" Jul 06 23:49:36.911809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.911796 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 23:49:36.912266 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.912240 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.912340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.912283 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3acadf3-310e-4f27-99ed-c62ee39567a1-serving-cert\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.916223 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.916202 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 23:49:36.916953 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.916922 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7kdf\" (UniqueName: \"kubernetes.io/projected/a3acadf3-310e-4f27-99ed-c62ee39567a1-kube-api-access-g7kdf\") pod \"console-operator-68b757fb67-lz94n\" (UID: \"a3acadf3-310e-4f27-99ed-c62ee39567a1\") " pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:36.917202 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.917184 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zn4h\" (UniqueName: \"kubernetes.io/projected/8e28fdd5-ea71-4ec9-a304-4dff04bf11f3-kube-api-access-5zn4h\") pod \"kube-storage-version-migrator-operator-846bf668f7-hbn4z\" (UID: \"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:36.921428 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.921406 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsml6\" (UniqueName: \"kubernetes.io/projected/1c1afb1b-4e1e-47c9-9279-35f4e103d5a7-kube-api-access-dsml6\") pod \"service-ca-operator-5d46d9ffd4-prg5s\" (UID: \"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:36.950997 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.950982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" Jul 06 23:49:36.983166 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:36.983147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" Jul 06 23:49:37.010632 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nldz\" (UniqueName: \"kubernetes.io/projected/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-kube-api-access-2nldz\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:37.010737 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:37.010737 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.010737 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-image-registry-private-configuration\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.010925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-bound-sa-token\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.010925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sngzr\" (UniqueName: \"kubernetes.io/projected/7b6d7f24-f100-4629-8acc-2c15f99632bd-kube-api-access-sngzr\") pod \"network-check-source-989f8b8dc-tqzsb\" (UID: \"7b6d7f24-f100-4629-8acc-2c15f99632bd\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" Jul 06 23:49:37.010925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-trusted-ca\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.010925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62f2w\" (UniqueName: \"kubernetes.io/projected/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-kube-api-access-62f2w\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:37.010925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-installation-pull-secrets\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.010925 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfmq\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-kube-api-access-8lfmq\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.011188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:37.011188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:37.011188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.010994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-ca-trust-extracted\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.011188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.011018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-certificates\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.011403 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.011293 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:37.011403 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.011326 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:49:37.011403 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.011362 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls podName:d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:37.511342226 +0000 UTC m=+42.161764152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-tqtxf" (UID: "d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:37.011403 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.011382 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls podName:49baa1b8-df31-46c9-afa4-b33c0aca5b2d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:37.511372777 +0000 UTC m=+42.161794703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-p82h4" (UID: "49baa1b8-df31-46c9-afa4-b33c0aca5b2d") : secret "samples-operator-tls" not found Jul 06 23:49:37.011740 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.011722 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:37.019971 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.019951 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62f2w\" (UniqueName: \"kubernetes.io/projected/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-kube-api-access-62f2w\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:37.020061 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.020046 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sngzr\" (UniqueName: \"kubernetes.io/projected/7b6d7f24-f100-4629-8acc-2c15f99632bd-kube-api-access-sngzr\") pod \"network-check-source-989f8b8dc-tqzsb\" (UID: \"7b6d7f24-f100-4629-8acc-2c15f99632bd\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" Jul 06 23:49:37.020188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.020166 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nldz\" (UniqueName: \"kubernetes.io/projected/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-kube-api-access-2nldz\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:37.043135 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.043111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" Jul 06 23:49:37.060307 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.060282 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" Jul 06 23:49:37.075266 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.074635 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:37.112089 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-ca-trust-extracted\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112206 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-certificates\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112259 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112309 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-image-registry-private-configuration\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112309 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-bound-sa-token\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112420 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-trusted-ca\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112420 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-installation-pull-secrets\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112420 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfmq\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-kube-api-access-8lfmq\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.112611 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.112480 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-ca-trust-extracted\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.113143 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.112828 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:49:37.113143 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.112856 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7d5b7dd4-88rkl: secret "image-registry-tls" not found Jul 06 23:49:37.113143 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.112909 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls podName:25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:37.612890775 +0000 UTC m=+42.263312700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls") pod "image-registry-6f7d5b7dd4-88rkl" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59") : secret "image-registry-tls" not found Jul 06 23:49:37.113143 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.113071 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-certificates\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.113878 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.113841 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-trusted-ca\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.115539 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.115515 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-image-registry-private-configuration\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.115737 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.115716 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-installation-pull-secrets\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.122412 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.122366 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfmq\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-kube-api-access-8lfmq\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.122648 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.122601 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-bound-sa-token\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.123458 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.123380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8"] Jul 06 23:49:37.128831 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:49:37.128734 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c1d9cf_b5e4_47b5_ae63_77ea95565764.slice/crio-96c6580b4ff8cd7b55a1f488e16305e9baf146ffa478eebeef03869181cb658f WatchSource:0}: Error finding container 96c6580b4ff8cd7b55a1f488e16305e9baf146ffa478eebeef03869181cb658f: Status 404 returned error can't find the container with id 96c6580b4ff8cd7b55a1f488e16305e9baf146ffa478eebeef03869181cb658f Jul 06 23:49:37.170767 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.170715 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" Jul 06 23:49:37.180778 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.177522 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xgj9q"] Jul 06 23:49:37.180778 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:49:37.180290 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd76a4e_600e_458e_adb3_801d5f30c4be.slice/crio-1f794c124314d1749421fe43f180d4fb2b1f14f7d2f33ca246aefa36139d1ee1 WatchSource:0}: Error finding container 1f794c124314d1749421fe43f180d4fb2b1f14f7d2f33ca246aefa36139d1ee1: Status 404 returned error can't find the container with id 1f794c124314d1749421fe43f180d4fb2b1f14f7d2f33ca246aefa36139d1ee1 Jul 06 23:49:37.223117 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.223083 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s"] Jul 06 23:49:37.226578 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:49:37.226099 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1afb1b_4e1e_47c9_9279_35f4e103d5a7.slice/crio-e0c726ee66871d8bed00d05e82884086bae084592ef57597c73be70b162b7b3e WatchSource:0}: Error finding container e0c726ee66871d8bed00d05e82884086bae084592ef57597c73be70b162b7b3e: Status 404 returned error can't find the container with id e0c726ee66871d8bed00d05e82884086bae084592ef57597c73be70b162b7b3e Jul 06 23:49:37.257087 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.256756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z"] Jul 06 23:49:37.261098 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:49:37.260984 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e28fdd5_ea71_4ec9_a304_4dff04bf11f3.slice/crio-9339672adad90df2c15a36b722e4ad226914ffdd88441a057142ea6491138544 WatchSource:0}: Error finding container 9339672adad90df2c15a36b722e4ad226914ffdd88441a057142ea6491138544: Status 404 returned error can't find the container with id 9339672adad90df2c15a36b722e4ad226914ffdd88441a057142ea6491138544 Jul 06 23:49:37.287200 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.287175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-lz94n"] Jul 06 23:49:37.290028 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:49:37.290005 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3acadf3_310e_4f27_99ed_c62ee39567a1.slice/crio-7d0e8628fa304f7eb19818a3427d079be502e6a8f0918802c007bb9d194fdfba WatchSource:0}: Error finding container 7d0e8628fa304f7eb19818a3427d079be502e6a8f0918802c007bb9d194fdfba: Status 404 returned error can't find the container with id 7d0e8628fa304f7eb19818a3427d079be502e6a8f0918802c007bb9d194fdfba Jul 06 23:49:37.314403 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.314372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:37.314515 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.314424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:37.314560 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.314547 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:38.31452924 +0000 UTC m=+42.964951164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : configmap references non-existent config key: service-ca.crt Jul 06 23:49:37.314605 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.314551 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:49:37.314605 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.314602 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:38.31458705 +0000 UTC m=+42.965008975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : secret "router-metrics-certs-default" not found Jul 06 23:49:37.333339 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.331900 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb"] Jul 06 23:49:37.336424 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:49:37.336400 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6d7f24_f100_4629_8acc_2c15f99632bd.slice/crio-2fa95162d920a124b66fc2c8b8889dc277071278c801d575fd521699a72f369e WatchSource:0}: Error finding container 2fa95162d920a124b66fc2c8b8889dc277071278c801d575fd521699a72f369e: Status 404 returned error can't find the container with id 2fa95162d920a124b66fc2c8b8889dc277071278c801d575fd521699a72f369e Jul 06 23:49:37.516107 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.516075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:37.516242 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.516115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:37.516300 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.516247 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:37.516300 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.516264 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:49:37.516402 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.516327 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls podName:d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:38.516305711 +0000 UTC m=+43.166727652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-tqtxf" (UID: "d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:37.516402 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.516348 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls podName:49baa1b8-df31-46c9-afa4-b33c0aca5b2d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:38.516337755 +0000 UTC m=+43.166759681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-p82h4" (UID: "49baa1b8-df31-46c9-afa4-b33c0aca5b2d") : secret "samples-operator-tls" not found Jul 06 23:49:37.616620 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:37.616588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:37.616962 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.616702 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:49:37.616962 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.616714 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7d5b7dd4-88rkl: secret "image-registry-tls" not found Jul 06 23:49:37.616962 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:37.616764 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls podName:25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:38.61674651 +0000 UTC m=+43.267168437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls") pod "image-registry-6f7d5b7dd4-88rkl" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59") : secret "image-registry-tls" not found Jul 06 23:49:38.009324 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.009240 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" event={"ID":"49c1d9cf-b5e4-47b5-ae63-77ea95565764","Type":"ContainerStarted","Data":"96c6580b4ff8cd7b55a1f488e16305e9baf146ffa478eebeef03869181cb658f"} Jul 06 23:49:38.010922 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.010890 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" event={"ID":"7b6d7f24-f100-4629-8acc-2c15f99632bd","Type":"ContainerStarted","Data":"2fa95162d920a124b66fc2c8b8889dc277071278c801d575fd521699a72f369e"} Jul 06 23:49:38.014753 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.012484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" event={"ID":"a3acadf3-310e-4f27-99ed-c62ee39567a1","Type":"ContainerStarted","Data":"7d0e8628fa304f7eb19818a3427d079be502e6a8f0918802c007bb9d194fdfba"} Jul 06 23:49:38.014753 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.014409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" event={"ID":"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3","Type":"ContainerStarted","Data":"9339672adad90df2c15a36b722e4ad226914ffdd88441a057142ea6491138544"} Jul 06 23:49:38.016900 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.016874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" event={"ID":"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7","Type":"ContainerStarted","Data":"e0c726ee66871d8bed00d05e82884086bae084592ef57597c73be70b162b7b3e"} Jul 06 23:49:38.018483 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.018460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" event={"ID":"bdd76a4e-600e-458e-adb3-801d5f30c4be","Type":"ContainerStarted","Data":"1f794c124314d1749421fe43f180d4fb2b1f14f7d2f33ca246aefa36139d1ee1"} Jul 06 23:49:38.322716 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.321894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:38.322716 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.321975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:38.322716 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.322142 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:49:38.322716 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.322205 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:40.32218763 +0000 UTC m=+44.972609560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : secret "router-metrics-certs-default" not found Jul 06 23:49:38.322716 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.322627 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:40.322609104 +0000 UTC m=+44.973031048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : configmap references non-existent config key: service-ca.crt Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.524058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.524109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.524141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.524363 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.524413 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls podName:49baa1b8-df31-46c9-afa4-b33c0aca5b2d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:40.524399803 +0000 UTC m=+45.174821727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-p82h4" (UID: "49baa1b8-df31-46c9-afa4-b33c0aca5b2d") : secret "samples-operator-tls" not found Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.524602 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.524749 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls podName:d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:40.524727706 +0000 UTC m=+45.175149648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-tqtxf" (UID: "d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.524767 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:49:38.524835 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.524812 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls podName:9cf530db-a95a-469f-9bde-41d882c205ab nodeName:}" failed. No retries permitted until 2026-07-06 23:49:46.524799875 +0000 UTC m=+51.175221801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls") pod "dns-default-jn4x7" (UID: "9cf530db-a95a-469f-9bde-41d882c205ab") : secret "dns-default-metrics-tls" not found Jul 06 23:49:38.625820 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:38.625728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:38.626171 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.625895 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:49:38.626171 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.625912 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7d5b7dd4-88rkl: secret "image-registry-tls" not found Jul 06 23:49:38.626171 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:38.625972 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls podName:25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:40.625950874 +0000 UTC m=+45.276372805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls") pod "image-registry-6f7d5b7dd4-88rkl" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59") : secret "image-registry-tls" not found Jul 06 23:49:40.342819 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:40.342773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:40.343193 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:40.342842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:40.343193 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.342944 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:44.342921512 +0000 UTC m=+48.993343450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : configmap references non-existent config key: service-ca.crt Jul 06 23:49:40.343193 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.342996 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:49:40.343193 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.343051 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:44.343036337 +0000 UTC m=+48.993458263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : secret "router-metrics-certs-default" not found Jul 06 23:49:40.544710 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:40.544670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:40.544878 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.544837 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:49:40.544878 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:40.544857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:40.545004 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.544902 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls podName:49baa1b8-df31-46c9-afa4-b33c0aca5b2d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:44.544884222 +0000 UTC m=+49.195306148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-p82h4" (UID: "49baa1b8-df31-46c9-afa4-b33c0aca5b2d") : secret "samples-operator-tls" not found Jul 06 23:49:40.545004 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.544961 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:40.545111 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.545008 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls podName:d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:44.5449965 +0000 UTC m=+49.195418424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-tqtxf" (UID: "d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:40.646260 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:40.646231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:40.646441 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.646408 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:49:40.646441 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.646429 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7d5b7dd4-88rkl: secret "image-registry-tls" not found Jul 06 23:49:40.646574 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:40.646510 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls podName:25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:44.646475014 +0000 UTC m=+49.296896951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls") pod "image-registry-6f7d5b7dd4-88rkl" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59") : secret "image-registry-tls" not found Jul 06 23:49:44.040131 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.039870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" event={"ID":"49c1d9cf-b5e4-47b5-ae63-77ea95565764","Type":"ContainerStarted","Data":"b9afaf33847254f041c3a2617a445308116dff2ec9b19d3fe74d149233a4afe0"} Jul 06 23:49:44.042176 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.041782 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" event={"ID":"7b6d7f24-f100-4629-8acc-2c15f99632bd","Type":"ContainerStarted","Data":"73f64ae8317d07bc74383d08dc6e2fa9a919975d93fcbf51efce4d25bd590d5c"} Jul 06 23:49:44.043598 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.043578 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/0.log" Jul 06 23:49:44.043704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.043615 2564 generic.go:356] "Generic (PLEG): container finished" podID="a3acadf3-310e-4f27-99ed-c62ee39567a1" containerID="36d326ca749ebcf70423a9fdff96a65e39e2342db25a3aba95e8ebea7a0b2cc6" exitCode=255 Jul 06 23:49:44.043861 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.043820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" event={"ID":"a3acadf3-310e-4f27-99ed-c62ee39567a1","Type":"ContainerDied","Data":"36d326ca749ebcf70423a9fdff96a65e39e2342db25a3aba95e8ebea7a0b2cc6"} Jul 06 23:49:44.043935 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.043908 2564 scope.go:117] "RemoveContainer" containerID="36d326ca749ebcf70423a9fdff96a65e39e2342db25a3aba95e8ebea7a0b2cc6" Jul 06 23:49:44.047869 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.047613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" event={"ID":"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3","Type":"ContainerStarted","Data":"e44d963479d8fd2edd315afa8f3d4f5f95dd9f7f83caa2a771016c184149c675"} Jul 06 23:49:44.049542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.049208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" event={"ID":"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7","Type":"ContainerStarted","Data":"39952200bd7152f809c3037c157b6978e387028768f99e89e39668402d4aa634"} Jul 06 23:49:44.050954 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.050935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" event={"ID":"bdd76a4e-600e-458e-adb3-801d5f30c4be","Type":"ContainerStarted","Data":"c70eff363cd2c1e4ac6bf9d519dbf925e52f9ee0c664987c81bc176ece65a940"} Jul 06 23:49:44.056929 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.056893 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-c9nx8" podStartSLOduration=1.7738327219999999 podStartE2EDuration="8.056881522s" totalImagesPullingTime="6.2830488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:44.055878209 +0000 UTC m=+48.706300171" watchObservedRunningTime="2026-07-06 23:49:44.056881522 +0000 UTC m=+48.707303469" Jul 06 23:49:44.105331 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.104601 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" podStartSLOduration=1.859229843 podStartE2EDuration="8.104582794s" totalImagesPullingTime="6.245352951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:44.07553613 +0000 UTC m=+48.725958093" watchObservedRunningTime="2026-07-06 23:49:44.104582794 +0000 UTC m=+48.755004742" Jul 06 23:49:44.132733 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.132687 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tqzsb" podStartSLOduration=1.950736346 podStartE2EDuration="8.132670257s" totalImagesPullingTime="6.181933911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:44.132523679 +0000 UTC m=+48.782945643" watchObservedRunningTime="2026-07-06 23:49:44.132670257 +0000 UTC m=+48.783092203" Jul 06 23:49:44.160649 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.160601 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" podStartSLOduration=1.834187689 podStartE2EDuration="8.160582964s" totalImagesPullingTime="6.326395275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:44.159029238 +0000 UTC m=+48.809451188" watchObservedRunningTime="2026-07-06 23:49:44.160582964 +0000 UTC m=+48.811004911" Jul 06 23:49:44.182424 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.181563 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" podStartSLOduration=1.8987769559999998 podStartE2EDuration="8.181546521s" totalImagesPullingTime="6.282769565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:44.178977346 +0000 UTC m=+48.829399295" watchObservedRunningTime="2026-07-06 23:49:44.181546521 +0000 UTC m=+48.831968469" Jul 06 23:49:44.381225 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.381149 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:44.381369 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.381240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:44.381369 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.381339 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:52.381316457 +0000 UTC m=+57.031738396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : configmap references non-existent config key: service-ca.crt Jul 06 23:49:44.381488 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.381401 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:49:44.381488 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.381465 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:52.381448439 +0000 UTC m=+57.031870367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : secret "router-metrics-certs-default" not found Jul 06 23:49:44.582786 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.582745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:44.582930 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.582795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:44.582982 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.582939 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:44.583021 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.582995 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:49:44.583021 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.583012 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls podName:d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:52.582991503 +0000 UTC m=+57.233413437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-tqtxf" (UID: "d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:44.583088 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.583053 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls podName:49baa1b8-df31-46c9-afa4-b33c0aca5b2d nodeName:}" failed. No retries permitted until 2026-07-06 23:49:52.58303626 +0000 UTC m=+57.233458192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-p82h4" (UID: "49baa1b8-df31-46c9-afa4-b33c0aca5b2d") : secret "samples-operator-tls" not found Jul 06 23:49:44.683474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:44.683447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:44.683632 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.683611 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:49:44.683632 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.683631 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7d5b7dd4-88rkl: secret "image-registry-tls" not found Jul 06 23:49:44.683735 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:44.683679 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls podName:25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:52.683663096 +0000 UTC m=+57.334085028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls") pod "image-registry-6f7d5b7dd4-88rkl" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59") : secret "image-registry-tls" not found Jul 06 23:49:45.056825 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.056754 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 06 23:49:45.057314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.057296 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/0.log" Jul 06 23:49:45.057377 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.057342 2564 generic.go:356] "Generic (PLEG): container finished" podID="a3acadf3-310e-4f27-99ed-c62ee39567a1" containerID="672bcf3db077ed4ad341022426e1714fd0b5ca6e57726ee97dc9196d2b15e842" exitCode=255 Jul 06 23:49:45.057456 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.057426 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" event={"ID":"a3acadf3-310e-4f27-99ed-c62ee39567a1","Type":"ContainerDied","Data":"672bcf3db077ed4ad341022426e1714fd0b5ca6e57726ee97dc9196d2b15e842"} Jul 06 23:49:45.057608 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.057521 2564 scope.go:117] "RemoveContainer" containerID="36d326ca749ebcf70423a9fdff96a65e39e2342db25a3aba95e8ebea7a0b2cc6" Jul 06 23:49:45.065540 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.065474 2564 scope.go:117] "RemoveContainer" containerID="672bcf3db077ed4ad341022426e1714fd0b5ca6e57726ee97dc9196d2b15e842" Jul 06 23:49:45.065757 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:45.065732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lz94n_openshift-console-operator(a3acadf3-310e-4f27-99ed-c62ee39567a1)\"" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" podUID="a3acadf3-310e-4f27-99ed-c62ee39567a1" Jul 06 23:49:45.551673 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.551638 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4wsvn"] Jul 06 23:49:45.566651 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.566629 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.568973 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.568948 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 23:49:45.569195 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.568948 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lr2cs\"" Jul 06 23:49:45.569300 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.569075 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 23:49:45.570020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.569998 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wsvn"] Jul 06 23:49:45.696420 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.696387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lx2\" (UniqueName: \"kubernetes.io/projected/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-api-access-z4lx2\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.696572 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.696439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-crio-socket\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.696623 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.696604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-data-volume\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.696680 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.696667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.696719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.696696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.797113 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.797084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-data-volume\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.797262 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.797179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.797262 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.797215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.797393 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:45.797302 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:49:45.797393 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.797313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lx2\" (UniqueName: \"kubernetes.io/projected/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-api-access-z4lx2\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.797393 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:45.797358 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:46.297338682 +0000 UTC m=+50.947760611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : configmap "kube-rbac-proxy" not found Jul 06 23:49:45.797393 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.797378 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-data-volume\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.797393 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.797390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-crio-socket\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.797664 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:45.797429 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:49:45.797664 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:45.797491 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:46.297475156 +0000 UTC m=+50.947897081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:49:45.797664 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.797586 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-crio-socket\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:45.805516 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:45.805443 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lx2\" (UniqueName: \"kubernetes.io/projected/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-api-access-z4lx2\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:46.062543 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:46.062460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 06 23:49:46.062956 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:46.062875 2564 scope.go:117] "RemoveContainer" containerID="672bcf3db077ed4ad341022426e1714fd0b5ca6e57726ee97dc9196d2b15e842" Jul 06 23:49:46.063087 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:46.063065 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lz94n_openshift-console-operator(a3acadf3-310e-4f27-99ed-c62ee39567a1)\"" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" podUID="a3acadf3-310e-4f27-99ed-c62ee39567a1" Jul 06 23:49:46.303225 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:46.303186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:46.303225 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:46.303229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:46.303421 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:46.303337 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:49:46.303421 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:46.303350 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:49:46.303421 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:46.303406 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:47.30339004 +0000 UTC m=+51.953811966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : configmap "kube-rbac-proxy" not found Jul 06 23:49:46.303421 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:46.303421 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:47.303413184 +0000 UTC m=+51.953835110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:49:46.605639 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:46.605603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:49:46.605828 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:46.605779 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:49:46.605898 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:46.605858 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls podName:9cf530db-a95a-469f-9bde-41d882c205ab nodeName:}" failed. No retries permitted until 2026-07-06 23:50:02.605839424 +0000 UTC m=+67.256261368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls") pod "dns-default-jn4x7" (UID: "9cf530db-a95a-469f-9bde-41d882c205ab") : secret "dns-default-metrics-tls" not found Jul 06 23:49:47.074817 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:47.074784 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:47.075167 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:47.074857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:47.075167 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:47.075156 2564 scope.go:117] "RemoveContainer" containerID="672bcf3db077ed4ad341022426e1714fd0b5ca6e57726ee97dc9196d2b15e842" Jul 06 23:49:47.075327 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:47.075309 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-lz94n_openshift-console-operator(a3acadf3-310e-4f27-99ed-c62ee39567a1)\"" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" podUID="a3acadf3-310e-4f27-99ed-c62ee39567a1" Jul 06 23:49:47.312134 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:47.312105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:47.312283 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:47.312140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:47.312283 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:47.312247 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:49:47.312363 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:47.312289 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:49:47.312363 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:47.312311 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:49.312298002 +0000 UTC m=+53.962719927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:49:47.312363 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:47.312342 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:49.312328945 +0000 UTC m=+53.962750870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : configmap "kube-rbac-proxy" not found Jul 06 23:49:48.141221 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:48.141192 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5kssz_f9e3d2fc-85d2-4071-a95b-9ca5be60a285/dns-node-resolver/0.log" Jul 06 23:49:49.328111 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:49.328078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:49.328491 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:49.328119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:49.328491 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:49.328199 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:49:49.328491 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:49.328246 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:49:49.328491 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:49.328267 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:53.32824949 +0000 UTC m=+57.978671432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : configmap "kube-rbac-proxy" not found Jul 06 23:49:49.328491 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:49.328300 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:49:53.328287208 +0000 UTC m=+57.978709134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:49:49.550825 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:49.550799 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9ds8_1ef88259-880c-4c27-a7c5-500f8b1f2698/node-ca/0.log" Jul 06 23:49:49.945267 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:49.945241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7kbvg_4529a4c7-0153-476f-bf94-566619bd5d22/serve-healthcheck-canary/0.log" Jul 06 23:49:50.941388 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:50.941357 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-hbn4z_8e28fdd5-ea71-4ec9-a304-4dff04bf11f3/kube-storage-version-migrator-operator/0.log" Jul 06 23:49:52.451044 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:52.450998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:52.451593 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:52.451090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:49:52.451593 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.451219 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:50:08.451188665 +0000 UTC m=+73.101610603 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : configmap references non-existent config key: service-ca.crt Jul 06 23:49:52.451593 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.451258 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 23:49:52.451593 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.451305 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs podName:beb31d50-ff75-48a0-afc9-42ce85ddf86d nodeName:}" failed. No retries permitted until 2026-07-06 23:50:08.451293805 +0000 UTC m=+73.101715730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs") pod "router-default-65b4d79d85-rmpz9" (UID: "beb31d50-ff75-48a0-afc9-42ce85ddf86d") : secret "router-metrics-certs-default" not found Jul 06 23:49:52.652450 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:52.652407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:49:52.652450 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:52.652455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:49:52.652724 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.652599 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:52.652724 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.652669 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls podName:d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:08.652653307 +0000 UTC m=+73.303075232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-tqtxf" (UID: "d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16") : secret "cluster-monitoring-operator-tls" not found Jul 06 23:49:52.652724 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.652692 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 23:49:52.652853 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.652762 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls podName:49baa1b8-df31-46c9-afa4-b33c0aca5b2d nodeName:}" failed. No retries permitted until 2026-07-06 23:50:08.652743845 +0000 UTC m=+73.303165791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-p82h4" (UID: "49baa1b8-df31-46c9-afa4-b33c0aca5b2d") : secret "samples-operator-tls" not found Jul 06 23:49:52.753439 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:52.753374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:49:52.753572 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.753541 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 23:49:52.753572 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.753565 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7d5b7dd4-88rkl: secret "image-registry-tls" not found Jul 06 23:49:52.753643 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:52.753612 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls podName:25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:08.753598436 +0000 UTC m=+73.404020361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls") pod "image-registry-6f7d5b7dd4-88rkl" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59") : secret "image-registry-tls" not found Jul 06 23:49:53.359053 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:53.359021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:53.359053 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:53.359059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:49:53.359256 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:53.359190 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:49:53.359256 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:53.359230 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:49:53.359321 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:53.359265 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:01.359249067 +0000 UTC m=+66.009670992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:49:53.359321 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:49:53.359284 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:01.359271609 +0000 UTC m=+66.009693534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : configmap "kube-rbac-proxy" not found Jul 06 23:49:56.979442 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:56.979415 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjgzw" Jul 06 23:49:57.799824 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:57.799796 2564 scope.go:117] "RemoveContainer" containerID="672bcf3db077ed4ad341022426e1714fd0b5ca6e57726ee97dc9196d2b15e842" Jul 06 23:49:58.102358 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:58.102288 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 06 23:49:58.102724 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:58.102387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" event={"ID":"a3acadf3-310e-4f27-99ed-c62ee39567a1","Type":"ContainerStarted","Data":"9d4ba0efd3f637f7fe2705fd5bb5fceda6949f2af3a58e40dc8300ca878ce712"} Jul 06 23:49:58.102724 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:58.102657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:49:58.123358 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:58.123314 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" podStartSLOduration=15.904379905 podStartE2EDuration="22.123302109s" totalImagesPullingTime="6.218922204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:49:58.122897544 +0000 UTC m=+62.773319533" watchObservedRunningTime="2026-07-06 23:49:58.123302109 +0000 UTC m=+62.773724055" Jul 06 23:49:58.468107 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:49:58.468075 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-lz94n" Jul 06 23:50:01.428622 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.428573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:50:01.428622 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.428626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:50:01.429160 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:50:01.428752 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:50:01.429160 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:50:01.428837 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls podName:75cc4fb0-79f4-45ab-98ed-bf1038399bc2 nodeName:}" failed. No retries permitted until 2026-07-06 23:50:17.428815493 +0000 UTC m=+82.079237423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wsvn" (UID: "75cc4fb0-79f4-45ab-98ed-bf1038399bc2") : secret "insights-runtime-extractor-tls" not found Jul 06 23:50:01.429160 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.429063 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:50:01.529960 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.529933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:50:01.530102 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.530010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:50:01.532320 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.532298 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:50:01.532578 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.532561 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbf9s\" (UniqueName: \"kubernetes.io/projected/edba5765-8d28-4002-9d52-de6a671e6178-kube-api-access-sbf9s\") pod \"network-check-target-mptrb\" (UID: \"edba5765-8d28-4002-9d52-de6a671e6178\") " pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:50:01.542877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.542855 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267abdc-57d6-410e-9b07-6fa8dfdd5ac7-metrics-certs\") pod \"network-metrics-daemon-5k58z\" (UID: \"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7\") " pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:50:01.631090 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.631057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:50:01.633893 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.633870 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:50:01.644064 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.644040 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606-original-pull-secret\") pod \"global-pull-secret-syncer-zlnnn\" (UID: \"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606\") " pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:50:01.715806 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.715743 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7lfhk\"" Jul 06 23:50:01.720571 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.720555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlnnn" Jul 06 23:50:01.723642 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.723622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:50:01.728902 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.728880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m58d\"" Jul 06 23:50:01.737823 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.737797 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5k58z" Jul 06 23:50:01.866115 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.866064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlnnn"] Jul 06 23:50:01.880385 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.880347 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mptrb"] Jul 06 23:50:01.882294 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:01.882265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedba5765_8d28_4002_9d52_de6a671e6178.slice/crio-a2a8714dfbcb1f861a3269e4838f150feff25b8d4528a12ad0ad26bc76064dcd WatchSource:0}: Error finding container a2a8714dfbcb1f861a3269e4838f150feff25b8d4528a12ad0ad26bc76064dcd: Status 404 returned error can't find the container with id a2a8714dfbcb1f861a3269e4838f150feff25b8d4528a12ad0ad26bc76064dcd Jul 06 23:50:01.897006 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:01.896980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5k58z"] Jul 06 23:50:01.899977 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:01.899956 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1267abdc_57d6_410e_9b07_6fa8dfdd5ac7.slice/crio-244686179cdf2d702d2d881ba51149fb30685125e4e1db121ef701bf9ee3aa54 WatchSource:0}: Error finding container 244686179cdf2d702d2d881ba51149fb30685125e4e1db121ef701bf9ee3aa54: Status 404 returned error can't find the container with id 244686179cdf2d702d2d881ba51149fb30685125e4e1db121ef701bf9ee3aa54 Jul 06 23:50:02.115480 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.115392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zlnnn" event={"ID":"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606","Type":"ContainerStarted","Data":"2288fd2d21f554416f18fe3f66e2f737107008f767eb0a4d1332e9ee6d8b16d3"} Jul 06 23:50:02.116484 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.116458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5k58z" event={"ID":"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7","Type":"ContainerStarted","Data":"244686179cdf2d702d2d881ba51149fb30685125e4e1db121ef701bf9ee3aa54"} Jul 06 23:50:02.117868 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.117843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mptrb" event={"ID":"edba5765-8d28-4002-9d52-de6a671e6178","Type":"ContainerStarted","Data":"bc0f47b8d3e79241c6f788aa27c76deca60bdd02f87922ab41bc686ac03f7510"} Jul 06 23:50:02.117983 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.117879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mptrb" event={"ID":"edba5765-8d28-4002-9d52-de6a671e6178","Type":"ContainerStarted","Data":"a2a8714dfbcb1f861a3269e4838f150feff25b8d4528a12ad0ad26bc76064dcd"} Jul 06 23:50:02.118040 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.117980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:50:02.133315 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.133274 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mptrb" podStartSLOduration=66.13326066 podStartE2EDuration="1m6.13326066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:02.132700137 +0000 UTC m=+66.783122084" watchObservedRunningTime="2026-07-06 23:50:02.13326066 +0000 UTC m=+66.783682607" Jul 06 23:50:02.641428 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.640840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:50:02.644050 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.644022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf530db-a95a-469f-9bde-41d882c205ab-metrics-tls\") pod \"dns-default-jn4x7\" (UID: \"9cf530db-a95a-469f-9bde-41d882c205ab\") " pod="openshift-dns/dns-default-jn4x7" Jul 06 23:50:02.871232 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.871201 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p99x7\"" Jul 06 23:50:02.878106 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:02.878070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jn4x7" Jul 06 23:50:03.084282 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:03.084203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jn4x7"] Jul 06 23:50:03.171656 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:03.171584 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf530db_a95a_469f_9bde_41d882c205ab.slice/crio-0fb59486bb438d4c1de5249a5ecf09f206e54bf7821e781130402bab7f9e95a1 WatchSource:0}: Error finding container 0fb59486bb438d4c1de5249a5ecf09f206e54bf7821e781130402bab7f9e95a1: Status 404 returned error can't find the container with id 0fb59486bb438d4c1de5249a5ecf09f206e54bf7821e781130402bab7f9e95a1 Jul 06 23:50:04.128526 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:04.128192 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5k58z" event={"ID":"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7","Type":"ContainerStarted","Data":"5a05807ca8ce7838c47d6aef59c4aa832b435a3141e9f92414ebd64bb0ce7ae9"} Jul 06 23:50:04.128526 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:04.128258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5k58z" event={"ID":"1267abdc-57d6-410e-9b07-6fa8dfdd5ac7","Type":"ContainerStarted","Data":"00be6d65e2f3f638fe65d641b51ffe45ace5aa37f8d33101842f4a0184cefdfc"} Jul 06 23:50:04.129675 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:04.129647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jn4x7" event={"ID":"9cf530db-a95a-469f-9bde-41d882c205ab","Type":"ContainerStarted","Data":"0fb59486bb438d4c1de5249a5ecf09f206e54bf7821e781130402bab7f9e95a1"} Jul 06 23:50:04.143718 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:04.143667 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5k58z" podStartSLOduration=66.85596781 podStartE2EDuration="1m8.143646926s" totalImagesPullingTime="1.287679116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:04.141922068 +0000 UTC m=+68.792344044" watchObservedRunningTime="2026-07-06 23:50:04.143646926 +0000 UTC m=+68.794068874" Jul 06 23:50:07.143307 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:07.143267 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zlnnn" event={"ID":"1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606","Type":"ContainerStarted","Data":"ace905f5e6c6d3edbeef778492cdc17298ace37c90ef0dfe311ec5096d12a8b0"} Jul 06 23:50:07.144963 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:07.144939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jn4x7" event={"ID":"9cf530db-a95a-469f-9bde-41d882c205ab","Type":"ContainerStarted","Data":"aa16ea422cacca0014a4aeebc934c2a1cabb4f95c4559e728aded85e8c991c1d"} Jul 06 23:50:07.145067 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:07.144989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jn4x7" event={"ID":"9cf530db-a95a-469f-9bde-41d882c205ab","Type":"ContainerStarted","Data":"2fc7c755a7359ece7fe97a7040616b8543f0c40b97725947756d1c14732d4a1f"} Jul 06 23:50:07.145104 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:07.145075 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jn4x7" Jul 06 23:50:07.158511 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:07.158421 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zlnnn" podStartSLOduration=66.805490828 podStartE2EDuration="1m11.158405348s" totalImagesPullingTime="4.35291452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:07.158000086 +0000 UTC m=+71.808422035" watchObservedRunningTime="2026-07-06 23:50:07.158405348 +0000 UTC m=+71.808827296" Jul 06 23:50:07.174461 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:07.174412 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jn4x7" podStartSLOduration=34.141830682 podStartE2EDuration="37.174395172s" totalImagesPullingTime="3.03256449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:07.173816938 +0000 UTC m=+71.824238886" watchObservedRunningTime="2026-07-06 23:50:07.174395172 +0000 UTC m=+71.824817122" Jul 06 23:50:08.491248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.491214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:08.491735 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.491263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:08.491952 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.491929 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb31d50-ff75-48a0-afc9-42ce85ddf86d-service-ca-bundle\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:08.493748 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.493729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb31d50-ff75-48a0-afc9-42ce85ddf86d-metrics-certs\") pod \"router-default-65b4d79d85-rmpz9\" (UID: \"beb31d50-ff75-48a0-afc9-42ce85ddf86d\") " pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:08.693415 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.693376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:50:08.693415 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.693419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:50:08.695956 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.695931 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/49baa1b8-df31-46c9-afa4-b33c0aca5b2d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-p82h4\" (UID: \"49baa1b8-df31-46c9-afa4-b33c0aca5b2d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:50:08.695956 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.695943 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-tqtxf\" (UID: \"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:50:08.703343 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.703325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-np7xr\"" Jul 06 23:50:08.711341 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.711326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" Jul 06 23:50:08.793197 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.793174 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-njfrp\"" Jul 06 23:50:08.794374 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.794354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:50:08.799538 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.799516 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"image-registry-6f7d5b7dd4-88rkl\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:50:08.801364 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.801345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:08.844155 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.840168 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf"] Jul 06 23:50:08.931647 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.931617 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65b4d79d85-rmpz9"] Jul 06 23:50:08.934260 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:08.934235 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb31d50_ff75_48a0_afc9_42ce85ddf86d.slice/crio-099d01732b11eabeda4b349777935127c43ac0d8a0f1aefa7f409ff9d333ebcb WatchSource:0}: Error finding container 099d01732b11eabeda4b349777935127c43ac0d8a0f1aefa7f409ff9d333ebcb: Status 404 returned error can't find the container with id 099d01732b11eabeda4b349777935127c43ac0d8a0f1aefa7f409ff9d333ebcb Jul 06 23:50:08.988947 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.988928 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hd56n\"" Jul 06 23:50:08.997626 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:08.997604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" Jul 06 23:50:09.028242 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.028184 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5vhf7\"" Jul 06 23:50:09.036847 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.036826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:50:09.135150 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.135119 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4"] Jul 06 23:50:09.154786 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.154709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" event={"ID":"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16","Type":"ContainerStarted","Data":"ba6752ef1793681f3cc54ef86e08c73fa90683b48f3e8988f81cad5e7a7b94c1"} Jul 06 23:50:09.156711 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.156678 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" event={"ID":"beb31d50-ff75-48a0-afc9-42ce85ddf86d","Type":"ContainerStarted","Data":"d702ca66ae361a0359e5d2d80afc29ce06879331c6e86acf6ef3bb566e5ce7e2"} Jul 06 23:50:09.156814 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.156746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" event={"ID":"beb31d50-ff75-48a0-afc9-42ce85ddf86d","Type":"ContainerStarted","Data":"099d01732b11eabeda4b349777935127c43ac0d8a0f1aefa7f409ff9d333ebcb"} Jul 06 23:50:09.174313 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.174270 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" podStartSLOduration=33.174254635 podStartE2EDuration="33.174254635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:09.17368294 +0000 UTC m=+73.824104908" watchObservedRunningTime="2026-07-06 23:50:09.174254635 +0000 UTC m=+73.824676693" Jul 06 23:50:09.181479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.181451 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f7d5b7dd4-88rkl"] Jul 06 23:50:09.183901 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:09.183876 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cac0b3_c00b_4ba0_8bcd_00ebf8fe8b59.slice/crio-51ed3422a4a21783cf8e09e10c21e75306ee4c71024165fbf850a2710f569112 WatchSource:0}: Error finding container 51ed3422a4a21783cf8e09e10c21e75306ee4c71024165fbf850a2710f569112: Status 404 returned error can't find the container with id 51ed3422a4a21783cf8e09e10c21e75306ee4c71024165fbf850a2710f569112 Jul 06 23:50:09.806226 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.806201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:09.806663 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:09.806312 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:10.075400 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:10.075330 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f7d5b7dd4-88rkl"] Jul 06 23:50:10.170100 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:10.170066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" event={"ID":"49baa1b8-df31-46c9-afa4-b33c0aca5b2d","Type":"ContainerStarted","Data":"d9ca7cde734fe9b14ce96814a147a2406fec9a65b49e8a1f8c2c49023d2835ae"} Jul 06 23:50:10.172020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:10.171991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" event={"ID":"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59","Type":"ContainerStarted","Data":"d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23"} Jul 06 23:50:10.172131 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:10.172038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" event={"ID":"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59","Type":"ContainerStarted","Data":"51ed3422a4a21783cf8e09e10c21e75306ee4c71024165fbf850a2710f569112"} Jul 06 23:50:10.172402 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:10.172349 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:10.173398 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:10.173375 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-65b4d79d85-rmpz9" Jul 06 23:50:10.212784 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:10.212732 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" podStartSLOduration=34.212712826 podStartE2EDuration="34.212712826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:10.210593168 +0000 UTC m=+74.861015131" watchObservedRunningTime="2026-07-06 23:50:10.212712826 +0000 UTC m=+74.863134772" Jul 06 23:50:11.178846 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:11.178799 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" event={"ID":"d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16","Type":"ContainerStarted","Data":"75a8e33e32c8a93c5ac116a0458d37bacf7412588689164343013d35460877d8"} Jul 06 23:50:11.179581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:11.179559 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:50:11.197434 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:11.197379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-tqtxf" podStartSLOduration=33.411650737 podStartE2EDuration="35.197359248s" totalImagesPullingTime="1.785708511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:11.194642376 +0000 UTC m=+75.845064324" watchObservedRunningTime="2026-07-06 23:50:11.197359248 +0000 UTC m=+75.847781195" Jul 06 23:50:12.184224 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:12.184182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" event={"ID":"49baa1b8-df31-46c9-afa4-b33c0aca5b2d","Type":"ContainerStarted","Data":"cc7a98a2b0fef601c0e5af978d098f40ee3afbbcd5c3341029a78004e86c85ad"} Jul 06 23:50:12.184723 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:12.184248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" event={"ID":"49baa1b8-df31-46c9-afa4-b33c0aca5b2d","Type":"ContainerStarted","Data":"1bea6893e740e38c5ac14d0903c16d9e424fc4139ee9a0e438dadfc166e9c8f9"} Jul 06 23:50:12.202834 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:12.202788 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-p82h4" podStartSLOduration=34.019315745 podStartE2EDuration="36.202771442s" totalImagesPullingTime="2.183455697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:12.201236369 +0000 UTC m=+76.851658315" watchObservedRunningTime="2026-07-06 23:50:12.202771442 +0000 UTC m=+76.853193389" Jul 06 23:50:17.149878 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:17.149756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jn4x7" Jul 06 23:50:17.459560 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:17.459470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:50:17.461779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:17.461759 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75cc4fb0-79f4-45ab-98ed-bf1038399bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wsvn\" (UID: \"75cc4fb0-79f4-45ab-98ed-bf1038399bc2\") " pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:50:17.679851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:17.679818 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lr2cs\"" Jul 06 23:50:17.687627 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:17.687606 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wsvn" Jul 06 23:50:17.814753 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:17.814725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wsvn"] Jul 06 23:50:17.817727 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:17.817699 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cc4fb0_79f4_45ab_98ed_bf1038399bc2.slice/crio-8e7cb566d1fa83df39eecace4a8b424124fce24b6e39ff336a562c7e79a65000 WatchSource:0}: Error finding container 8e7cb566d1fa83df39eecace4a8b424124fce24b6e39ff336a562c7e79a65000: Status 404 returned error can't find the container with id 8e7cb566d1fa83df39eecace4a8b424124fce24b6e39ff336a562c7e79a65000 Jul 06 23:50:18.206742 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:18.206708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wsvn" event={"ID":"75cc4fb0-79f4-45ab-98ed-bf1038399bc2","Type":"ContainerStarted","Data":"8e7cb566d1fa83df39eecace4a8b424124fce24b6e39ff336a562c7e79a65000"} Jul 06 23:50:22.225288 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:22.225245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wsvn" event={"ID":"75cc4fb0-79f4-45ab-98ed-bf1038399bc2","Type":"ContainerStarted","Data":"a625804c28e0c731a508fd4d2b3f2875cdb865b2fd88ba9b3a3443d0dcd0913e"} Jul 06 23:50:23.231672 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:23.231632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wsvn" event={"ID":"75cc4fb0-79f4-45ab-98ed-bf1038399bc2","Type":"ContainerStarted","Data":"4aed6265192291e95f1b451c3150e8144caf7e168893682fe012c94c17878922"} Jul 06 23:50:24.963121 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.963092 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64c65fbd4c-r7kvf"] Jul 06 23:50:24.966188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.966166 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:24.968923 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.968875 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 23:50:24.969035 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.968940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 23:50:24.969107 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.969095 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 23:50:24.969596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.969573 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 23:50:24.969596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.969583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m6b92\"" Jul 06 23:50:24.969821 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.969625 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 23:50:24.970084 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.970063 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 23:50:24.970200 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.970083 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 23:50:24.975291 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.975270 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 23:50:24.978290 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:24.978268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c65fbd4c-r7kvf"] Jul 06 23:50:25.021248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.021221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-oauth-serving-cert\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.021352 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.021254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkzq\" (UniqueName: \"kubernetes.io/projected/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-kube-api-access-qbkzq\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.021352 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.021281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-config\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.021352 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.021301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-serving-cert\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.021352 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.021344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-oauth-config\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.021543 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.021361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-service-ca\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.021543 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.021377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-trusted-ca-bundle\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.121923 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.121842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-oauth-serving-cert\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.121923 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.121882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkzq\" (UniqueName: \"kubernetes.io/projected/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-kube-api-access-qbkzq\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.121923 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.121914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-config\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.122115 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122028 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-serving-cert\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.122115 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-oauth-config\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.122115 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-service-ca\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.122232 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-trusted-ca-bundle\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.122715 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122672 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-config\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.122715 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122672 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-oauth-serving-cert\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.123036 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122873 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-service-ca\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.123036 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.122957 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-trusted-ca-bundle\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.124609 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.124592 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-serving-cert\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.124679 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.124627 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-oauth-config\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.129428 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.129410 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkzq\" (UniqueName: \"kubernetes.io/projected/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-kube-api-access-qbkzq\") pod \"console-64c65fbd4c-r7kvf\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.241387 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.241354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wsvn" event={"ID":"75cc4fb0-79f4-45ab-98ed-bf1038399bc2","Type":"ContainerStarted","Data":"ee57a4720c5eadde1b4e64dca7dcaa6c77015fa3f0972d5de97cc8ab210b4d79"} Jul 06 23:50:25.260708 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.260659 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4wsvn" podStartSLOduration=33.332809064 podStartE2EDuration="40.260644443s" totalImagesPullingTime="6.927835379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:49:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:25.259656144 +0000 UTC m=+89.910078112" watchObservedRunningTime="2026-07-06 23:50:25.260644443 +0000 UTC m=+89.911066389" Jul 06 23:50:25.276036 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.276013 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:25.406520 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:25.406475 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c65fbd4c-r7kvf"] Jul 06 23:50:25.409159 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:25.409136 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c8c166_0b43_4994_ae7b_dad6f83a91e1.slice/crio-a36df6272aabab0e1244e6a26919d29a1bfca43bf9cfde74a16d3298710bddfe WatchSource:0}: Error finding container a36df6272aabab0e1244e6a26919d29a1bfca43bf9cfde74a16d3298710bddfe: Status 404 returned error can't find the container with id a36df6272aabab0e1244e6a26919d29a1bfca43bf9cfde74a16d3298710bddfe Jul 06 23:50:26.245432 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:26.245393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c65fbd4c-r7kvf" event={"ID":"f2c8c166-0b43-4994-ae7b-dad6f83a91e1","Type":"ContainerStarted","Data":"a36df6272aabab0e1244e6a26919d29a1bfca43bf9cfde74a16d3298710bddfe"} Jul 06 23:50:28.255006 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:28.254970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c65fbd4c-r7kvf" event={"ID":"f2c8c166-0b43-4994-ae7b-dad6f83a91e1","Type":"ContainerStarted","Data":"1358b96862f3fd62e89f891f0e91dde2626cb32ce68b4766c4683a57e372c808"} Jul 06 23:50:28.273138 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:28.273087 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64c65fbd4c-r7kvf" podStartSLOduration=1.562057941 podStartE2EDuration="4.273073362s" totalImagesPullingTime="2.711015421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:28.272934312 +0000 UTC m=+92.923356260" watchObservedRunningTime="2026-07-06 23:50:28.273073362 +0000 UTC m=+92.923495310" Jul 06 23:50:31.186361 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:31.186335 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:50:33.123302 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.123272 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mptrb" Jul 06 23:50:33.588635 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.588600 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fbc9q"] Jul 06 23:50:33.592086 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.592070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.594558 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.594537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 23:50:33.594914 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.594888 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 23:50:33.594914 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.594905 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 23:50:33.595162 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.595143 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 23:50:33.595392 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.595378 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j6ctg\"" Jul 06 23:50:33.687597 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-wtmp\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687707 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-textfile\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687707 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwwd\" (UniqueName: \"kubernetes.io/projected/75cceadc-045e-461f-b931-ba21d5855cc0-kube-api-access-lgwwd\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687789 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687789 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-sys\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-root\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-tls\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.687972 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.687914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75cceadc-045e-461f-b931-ba21d5855cc0-metrics-client-ca\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789118 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwwd\" (UniqueName: \"kubernetes.io/projected/75cceadc-045e-461f-b931-ba21d5855cc0-kube-api-access-lgwwd\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-sys\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-root\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-tls\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789247 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-sys\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75cceadc-045e-461f-b931-ba21d5855cc0-metrics-client-ca\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789305 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-root\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-wtmp\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789474 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-textfile\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789724 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789577 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-wtmp\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789775 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-textfile\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789829 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789808 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.789872 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.789814 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75cceadc-045e-461f-b931-ba21d5855cc0-metrics-client-ca\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.791718 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.791694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-tls\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.791718 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.791708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75cceadc-045e-461f-b931-ba21d5855cc0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.796986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.796967 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwwd\" (UniqueName: \"kubernetes.io/projected/75cceadc-045e-461f-b931-ba21d5855cc0-kube-api-access-lgwwd\") pod \"node-exporter-fbc9q\" (UID: \"75cceadc-045e-461f-b931-ba21d5855cc0\") " pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.901741 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:33.901711 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbc9q" Jul 06 23:50:33.913229 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:33.913195 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cceadc_045e_461f_b931_ba21d5855cc0.slice/crio-37c9af4c4e8cad0539b54cda5a42d60318a1c9c07adc1a9540bfad40d12f7987 WatchSource:0}: Error finding container 37c9af4c4e8cad0539b54cda5a42d60318a1c9c07adc1a9540bfad40d12f7987: Status 404 returned error can't find the container with id 37c9af4c4e8cad0539b54cda5a42d60318a1c9c07adc1a9540bfad40d12f7987 Jul 06 23:50:34.278782 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:34.278699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbc9q" event={"ID":"75cceadc-045e-461f-b931-ba21d5855cc0","Type":"ContainerStarted","Data":"37c9af4c4e8cad0539b54cda5a42d60318a1c9c07adc1a9540bfad40d12f7987"} Jul 06 23:50:35.276806 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:35.276722 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:35.276806 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:35.276789 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:35.282066 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:35.282041 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:35.284441 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:35.284397 2564 generic.go:356] "Generic (PLEG): container finished" podID="75cceadc-045e-461f-b931-ba21d5855cc0" containerID="6573f15ffc328146ea7ce86c84044d7239ef0a3c704e9d27900e6e978b74b04b" exitCode=0 Jul 06 23:50:35.284590 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:35.284445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbc9q" event={"ID":"75cceadc-045e-461f-b931-ba21d5855cc0","Type":"ContainerDied","Data":"6573f15ffc328146ea7ce86c84044d7239ef0a3c704e9d27900e6e978b74b04b"} Jul 06 23:50:35.289291 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:35.289268 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:50:36.203228 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.203164 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" podUID="25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" containerName="registry" containerID="cri-o://d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23" gracePeriod=30 Jul 06 23:50:36.290425 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.290391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbc9q" event={"ID":"75cceadc-045e-461f-b931-ba21d5855cc0","Type":"ContainerStarted","Data":"d8d0f1a2aa069d74c25e87a256d41d2ad27fdf63b465899ad846d2d249a0e9a9"} Jul 06 23:50:36.290741 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.290448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbc9q" event={"ID":"75cceadc-045e-461f-b931-ba21d5855cc0","Type":"ContainerStarted","Data":"bee2c81d5b53d72b94ebe875dec5b1e52fff370807d2c1f3abe7855c4e0f341b"} Jul 06 23:50:36.311234 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.311191 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fbc9q" podStartSLOduration=2.188049003 podStartE2EDuration="3.31117732s" totalImagesPullingTime="1.053516256s" totalInitContainerRuntime="69.612061ms" isStatefulPod=true podCreationTimestamp="2026-07-06 23:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:36.310718792 +0000 UTC m=+100.961140758" watchObservedRunningTime="2026-07-06 23:50:36.31117732 +0000 UTC m=+100.961599267" Jul 06 23:50:36.440382 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.440359 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:50:36.514521 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514447 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-ca-trust-extracted\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.514521 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514486 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-image-registry-private-configuration\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.514712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514538 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-installation-pull-secrets\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.514712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514571 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-trusted-ca\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.514712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514589 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-bound-sa-token\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.514712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514604 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-certificates\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.514924 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.514924 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.514834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lfmq\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-kube-api-access-8lfmq\") pod \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\" (UID: \"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59\") " Jul 06 23:50:36.515097 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.515071 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:50:36.515097 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.515085 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:50:36.517459 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.517430 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:36.517583 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.517453 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:50:36.517583 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.517566 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-kube-api-access-8lfmq" (OuterVolumeSpecName: "kube-api-access-8lfmq") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "kube-api-access-8lfmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:36.517706 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.517660 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:36.517706 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.517676 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:50:36.524711 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.524684 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" (UID: "25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:36.549127 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.549101 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c68854b46-x79cd"] Jul 06 23:50:36.549358 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.549347 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" containerName="registry" Jul 06 23:50:36.549398 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.549360 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" containerName="registry" Jul 06 23:50:36.549435 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.549426 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" containerName="registry" Jul 06 23:50:36.554088 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.554075 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.556179 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.556146 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 23:50:36.556179 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.556173 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 23:50:36.556369 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.556354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fs6bfp55g8o02\"" Jul 06 23:50:36.556436 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.556416 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9ss2h\"" Jul 06 23:50:36.556542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.556417 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 23:50:36.556655 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.556534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 23:50:36.556724 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.556709 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 23:50:36.563485 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.563467 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c68854b46-x79cd"] Jul 06 23:50:36.616344 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616344 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616569 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616569 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616569 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4459g\" (UniqueName: \"kubernetes.io/projected/6442b11f-8354-4648-9b52-450a281f6033-kube-api-access-4459g\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616569 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6442b11f-8354-4648-9b52-450a281f6033-metrics-client-ca\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616569 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-grpc-tls\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-tls\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616643 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-installation-pull-secrets\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616652 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-trusted-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616661 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-bound-sa-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616669 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-certificates\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616677 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-registry-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616686 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfmq\" (UniqueName: \"kubernetes.io/projected/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-kube-api-access-8lfmq\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616695 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-ca-trust-extracted\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.616719 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.616704 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59-image-registry-private-configuration\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:50:36.717843 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.717781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4459g\" (UniqueName: \"kubernetes.io/projected/6442b11f-8354-4648-9b52-450a281f6033-kube-api-access-4459g\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718010 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.717848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6442b11f-8354-4648-9b52-450a281f6033-metrics-client-ca\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718010 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.717881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-grpc-tls\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718010 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.717919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-tls\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718010 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.717972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718200 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.718011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718200 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.718129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718200 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.718166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.718944 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.718917 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6442b11f-8354-4648-9b52-450a281f6033-metrics-client-ca\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.720681 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.720662 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.720801 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.720782 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-grpc-tls\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.721043 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.721024 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.721181 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.721157 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.721360 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.721341 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-tls\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.721551 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.721538 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6442b11f-8354-4648-9b52-450a281f6033-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.724894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.724869 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4459g\" (UniqueName: \"kubernetes.io/projected/6442b11f-8354-4648-9b52-450a281f6033-kube-api-access-4459g\") pod \"thanos-querier-7c68854b46-x79cd\" (UID: \"6442b11f-8354-4648-9b52-450a281f6033\") " pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:36.863378 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:36.863314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:37.000416 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.000394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c68854b46-x79cd"] Jul 06 23:50:37.002843 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:37.002815 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6442b11f_8354_4648_9b52_450a281f6033.slice/crio-4f01d28a1f8e575181815341a8ff8462e1f7ffbdfdc27a5e52f675f239d6e3f8 WatchSource:0}: Error finding container 4f01d28a1f8e575181815341a8ff8462e1f7ffbdfdc27a5e52f675f239d6e3f8: Status 404 returned error can't find the container with id 4f01d28a1f8e575181815341a8ff8462e1f7ffbdfdc27a5e52f675f239d6e3f8 Jul 06 23:50:37.295306 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.295274 2564 generic.go:356] "Generic (PLEG): container finished" podID="25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" containerID="d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23" exitCode=0 Jul 06 23:50:37.295741 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.295317 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" Jul 06 23:50:37.295741 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.295360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" event={"ID":"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59","Type":"ContainerDied","Data":"d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23"} Jul 06 23:50:37.295741 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.295408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f7d5b7dd4-88rkl" event={"ID":"25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59","Type":"ContainerDied","Data":"51ed3422a4a21783cf8e09e10c21e75306ee4c71024165fbf850a2710f569112"} Jul 06 23:50:37.295741 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.295437 2564 scope.go:117] "RemoveContainer" containerID="d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23" Jul 06 23:50:37.296887 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.296866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" event={"ID":"6442b11f-8354-4648-9b52-450a281f6033","Type":"ContainerStarted","Data":"4f01d28a1f8e575181815341a8ff8462e1f7ffbdfdc27a5e52f675f239d6e3f8"} Jul 06 23:50:37.303759 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.303742 2564 scope.go:117] "RemoveContainer" containerID="d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23" Jul 06 23:50:37.304032 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:50:37.304015 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23\": container with ID starting with d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23 not found: ID does not exist" containerID="d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23" Jul 06 23:50:37.304077 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.304040 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23"} err="failed to get container status \"d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23\": rpc error: code = NotFound desc = could not find container \"d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23\": container with ID starting with d5919d697618ac4a3bcf502a525a824ca301429b0fc6e183d7a5fe531c89eb23 not found: ID does not exist" Jul 06 23:50:37.315354 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.315334 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f7d5b7dd4-88rkl"] Jul 06 23:50:37.320381 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.320362 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f7d5b7dd4-88rkl"] Jul 06 23:50:37.806784 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.806744 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59" path="/var/lib/kubelet/pods/25cac0b3-c00b-4ba0-8bcd-00ebf8fe8b59/volumes" Jul 06 23:50:37.968632 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.968587 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bd45d5848-ssflm"] Jul 06 23:50:37.983009 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.982984 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bd45d5848-ssflm"] Jul 06 23:50:37.983156 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.983089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:37.986139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.985933 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-h7z8x\"" Jul 06 23:50:37.986139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.985933 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 23:50:37.986139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.986004 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 23:50:37.986139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.986016 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 23:50:37.986139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.986034 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 23:50:37.986139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:37.986006 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-finp42ffo7e9o\"" Jul 06 23:50:38.029573 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.029545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-secret-metrics-server-tls\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.029714 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.029606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.029714 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.029652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-audit-log\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.029714 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.029699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vxw\" (UniqueName: \"kubernetes.io/projected/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-kube-api-access-x4vxw\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.029855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.029742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-metrics-server-audit-profiles\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.029855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.029778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-secret-metrics-server-client-certs\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.029855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.029808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-client-ca-bundle\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.130619 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-audit-log\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.130619 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vxw\" (UniqueName: \"kubernetes.io/projected/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-kube-api-access-x4vxw\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.130619 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-metrics-server-audit-profiles\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.131003 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-secret-metrics-server-client-certs\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.131003 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-client-ca-bundle\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.131003 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-secret-metrics-server-tls\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.131003 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.131003 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.130961 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-audit-log\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.131652 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.131534 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.132366 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.132324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-metrics-server-audit-profiles\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.134029 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.134005 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-secret-metrics-server-client-certs\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.134423 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.134400 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-client-ca-bundle\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.135087 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.135064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-secret-metrics-server-tls\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.140824 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.140805 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vxw\" (UniqueName: \"kubernetes.io/projected/7fcf0797-a1e3-4ae2-949f-3a96fcf52453-kube-api-access-x4vxw\") pod \"metrics-server-bd45d5848-ssflm\" (UID: \"7fcf0797-a1e3-4ae2-949f-3a96fcf52453\") " pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.298768 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.293147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:38.393251 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.391671 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7"] Jul 06 23:50:38.427788 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.427658 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7"] Jul 06 23:50:38.427788 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.427778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:38.431180 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.431150 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 23:50:38.431558 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.431156 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-94jjk\"" Jul 06 23:50:38.486113 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.486087 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bd45d5848-ssflm"] Jul 06 23:50:38.490768 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:38.490544 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcf0797_a1e3_4ae2_949f_3a96fcf52453.slice/crio-4cc4fb2dba92785fabc8bf0b0057560c4adc0f7897858a9c9261099b6421f345 WatchSource:0}: Error finding container 4cc4fb2dba92785fabc8bf0b0057560c4adc0f7897858a9c9261099b6421f345: Status 404 returned error can't find the container with id 4cc4fb2dba92785fabc8bf0b0057560c4adc0f7897858a9c9261099b6421f345 Jul 06 23:50:38.534207 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.534181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552lh\" (UniqueName: \"kubernetes.io/projected/dbf032d3-63ac-41e3-b3ea-1145e00556b1-kube-api-access-552lh\") pod \"monitoring-plugin-d5dcff7cf-pl2c7\" (UID: \"dbf032d3-63ac-41e3-b3ea-1145e00556b1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:38.534300 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.534231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dbf032d3-63ac-41e3-b3ea-1145e00556b1-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-pl2c7\" (UID: \"dbf032d3-63ac-41e3-b3ea-1145e00556b1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:38.635650 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.635621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552lh\" (UniqueName: \"kubernetes.io/projected/dbf032d3-63ac-41e3-b3ea-1145e00556b1-kube-api-access-552lh\") pod \"monitoring-plugin-d5dcff7cf-pl2c7\" (UID: \"dbf032d3-63ac-41e3-b3ea-1145e00556b1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:38.635793 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.635676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dbf032d3-63ac-41e3-b3ea-1145e00556b1-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-pl2c7\" (UID: \"dbf032d3-63ac-41e3-b3ea-1145e00556b1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:38.638254 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.638228 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dbf032d3-63ac-41e3-b3ea-1145e00556b1-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-pl2c7\" (UID: \"dbf032d3-63ac-41e3-b3ea-1145e00556b1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:38.652377 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.652324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552lh\" (UniqueName: \"kubernetes.io/projected/dbf032d3-63ac-41e3-b3ea-1145e00556b1-kube-api-access-552lh\") pod \"monitoring-plugin-d5dcff7cf-pl2c7\" (UID: \"dbf032d3-63ac-41e3-b3ea-1145e00556b1\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:38.742260 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:38.742211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:39.310372 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:39.310342 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" event={"ID":"7fcf0797-a1e3-4ae2-949f-3a96fcf52453","Type":"ContainerStarted","Data":"4cc4fb2dba92785fabc8bf0b0057560c4adc0f7897858a9c9261099b6421f345"} Jul 06 23:50:39.378780 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:39.378752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7"] Jul 06 23:50:39.382516 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:39.382477 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf032d3_63ac_41e3_b3ea_1145e00556b1.slice/crio-2d5b9b862aa4d88ab471b2770c4a61eba019e0278c6d3685a2a988bb183383ac WatchSource:0}: Error finding container 2d5b9b862aa4d88ab471b2770c4a61eba019e0278c6d3685a2a988bb183383ac: Status 404 returned error can't find the container with id 2d5b9b862aa4d88ab471b2770c4a61eba019e0278c6d3685a2a988bb183383ac Jul 06 23:50:39.997551 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:39.997509 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:50:40.003563 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.003516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.006117 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.006089 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 23:50:40.006265 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.006236 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 23:50:40.006540 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.006521 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 23:50:40.006654 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.006634 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 23:50:40.006774 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.006759 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 23:50:40.006888 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.006872 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 23:50:40.007074 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007055 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 23:50:40.007186 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007098 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 23:50:40.007240 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007219 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 23:50:40.007344 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 23:50:40.007489 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007472 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 23:50:40.007596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9g4betd0p8g1n\"" Jul 06 23:50:40.007596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007586 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fkcqz\"" Jul 06 23:50:40.007699 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.007610 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 23:50:40.010587 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.010568 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 23:50:40.014049 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.013899 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 23:50:40.029233 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.029203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:50:40.048421 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048588 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048588 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048588 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048734 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048734 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048734 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048734 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l879x\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-kube-api-access-l879x\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.048880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.049912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.049912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.049912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.049912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.048986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.049912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.049024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.049912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.049050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.049912 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.049078 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150060 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150028 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l879x\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-kube-api-access-l879x\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.150580 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.151154 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.151154 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.151154 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.151154 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150716 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.151154 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.150723 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.153212 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.152697 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.153212 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.152851 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.153321 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.153279 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.153765 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.153736 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.154191 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.154165 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.155602 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.155103 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.155602 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.155489 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.155602 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.155557 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.156076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.156032 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.156165 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.156116 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.156631 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.156603 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.157826 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.157735 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.157826 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.157737 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.158187 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.158164 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.158571 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.158549 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.158690 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.158616 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.159617 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.159597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.164088 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.164067 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l879x\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-kube-api-access-l879x\") pod \"prometheus-k8s-0\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.317211 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.317122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:40.318408 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.318360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" event={"ID":"6442b11f-8354-4648-9b52-450a281f6033","Type":"ContainerStarted","Data":"435ca250a117a9d185c56a82dc70354da6922985638ad188ed13ce033f0ed100"} Jul 06 23:50:40.318566 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.318527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" event={"ID":"6442b11f-8354-4648-9b52-450a281f6033","Type":"ContainerStarted","Data":"1189b474cd5de9c41cd2dcee118cac1193d95fdf87378793ef6e83a588167b77"} Jul 06 23:50:40.318633 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.318577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" event={"ID":"6442b11f-8354-4648-9b52-450a281f6033","Type":"ContainerStarted","Data":"e625a98c74dd4716a03a76066a92ded5e42fcd7538d103ed546b129f00ed8ab9"} Jul 06 23:50:40.320486 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:40.320459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" event={"ID":"dbf032d3-63ac-41e3-b3ea-1145e00556b1","Type":"ContainerStarted","Data":"2d5b9b862aa4d88ab471b2770c4a61eba019e0278c6d3685a2a988bb183383ac"} Jul 06 23:50:41.105002 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.104967 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:50:41.108402 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:50:41.108376 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb217e32_0f36_4d1c_a440_a9df49ffcfd8.slice/crio-58cfbc12e8bbf695cdb053e01a62490a8600d3880b897c971c382b8c053b5b54 WatchSource:0}: Error finding container 58cfbc12e8bbf695cdb053e01a62490a8600d3880b897c971c382b8c053b5b54: Status 404 returned error can't find the container with id 58cfbc12e8bbf695cdb053e01a62490a8600d3880b897c971c382b8c053b5b54 Jul 06 23:50:41.328121 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.327626 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" event={"ID":"dbf032d3-63ac-41e3-b3ea-1145e00556b1","Type":"ContainerStarted","Data":"50701a25cb30530de50ec10e97e7f3ec141e7ae15e8d2f2c725583cc94a647a1"} Jul 06 23:50:41.328121 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.327796 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:41.334172 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.334141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerStarted","Data":"58cfbc12e8bbf695cdb053e01a62490a8600d3880b897c971c382b8c053b5b54"} Jul 06 23:50:41.336528 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.336479 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" Jul 06 23:50:41.337855 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.337748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" event={"ID":"7fcf0797-a1e3-4ae2-949f-3a96fcf52453","Type":"ContainerStarted","Data":"ab8d25a232621782a2be83125a1ecb018277e462aac977d13cbd1295604bf804"} Jul 06 23:50:41.362258 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.362213 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" podStartSLOduration=1.895931426 podStartE2EDuration="4.362197509s" totalImagesPullingTime="2.466266083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:41.361107945 +0000 UTC m=+106.011529910" watchObservedRunningTime="2026-07-06 23:50:41.362197509 +0000 UTC m=+106.012619456" Jul 06 23:50:41.362531 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:41.362346 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-pl2c7" podStartSLOduration=1.7852653090000001 podStartE2EDuration="3.362341325s" totalImagesPullingTime="1.577076016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:41.345585825 +0000 UTC m=+105.996007798" watchObservedRunningTime="2026-07-06 23:50:41.362341325 +0000 UTC m=+106.012763273" Jul 06 23:50:42.343287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:42.343251 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" exitCode=0 Jul 06 23:50:42.343707 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:42.343340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13"} Jul 06 23:50:42.346767 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:42.346743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" event={"ID":"6442b11f-8354-4648-9b52-450a281f6033","Type":"ContainerStarted","Data":"2af5ba074b34d8317e6e26a05d1236aef11ae943f9378fbe6ce7ec01e7cb0fea"} Jul 06 23:50:42.346851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:42.346816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" event={"ID":"6442b11f-8354-4648-9b52-450a281f6033","Type":"ContainerStarted","Data":"3653cae9aab5c1388c0f7332f2b04ceac10a82ce435568c417194c790e263a8d"} Jul 06 23:50:42.346900 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:42.346858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" event={"ID":"6442b11f-8354-4648-9b52-450a281f6033","Type":"ContainerStarted","Data":"78cc4294044c3703bec8fdd40be294c64bfc50e2199ef09ce2790f4cdb86dd36"} Jul 06 23:50:42.396470 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:42.396383 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" podStartSLOduration=2.220090455 podStartE2EDuration="6.396363637s" totalImagesPullingTime="4.176273182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:42.395208854 +0000 UTC m=+107.045630827" watchObservedRunningTime="2026-07-06 23:50:42.396363637 +0000 UTC m=+107.046785587" Jul 06 23:50:43.354404 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:43.354374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:44.364113 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:44.364083 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c68854b46-x79cd" Jul 06 23:50:46.370129 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:46.370042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerStarted","Data":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} Jul 06 23:50:46.370547 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:46.370130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerStarted","Data":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} Jul 06 23:50:46.370547 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:46.370161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerStarted","Data":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} Jul 06 23:50:46.370547 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:46.370185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerStarted","Data":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} Jul 06 23:50:46.370547 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:46.370209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerStarted","Data":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} Jul 06 23:50:46.370547 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:46.370249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerStarted","Data":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} Jul 06 23:50:46.398861 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:46.398816 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.060217333 podStartE2EDuration="7.398799698s" totalImagesPullingTime="4.320486309s" totalInitContainerRuntime="18.096056ms" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:46.396120762 +0000 UTC m=+111.046542708" watchObservedRunningTime="2026-07-06 23:50:46.398799698 +0000 UTC m=+111.049221645" Jul 06 23:50:50.318540 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:50.318471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:50:55.326246 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:55.326218 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c65fbd4c-r7kvf"] Jul 06 23:50:55.405911 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:55.405862 2564 generic.go:356] "Generic (PLEG): container finished" podID="1c1afb1b-4e1e-47c9-9279-35f4e103d5a7" containerID="39952200bd7152f809c3037c157b6978e387028768f99e89e39668402d4aa634" exitCode=0 Jul 06 23:50:55.406090 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:55.405929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" event={"ID":"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7","Type":"ContainerDied","Data":"39952200bd7152f809c3037c157b6978e387028768f99e89e39668402d4aa634"} Jul 06 23:50:55.406312 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:55.406295 2564 scope.go:117] "RemoveContainer" containerID="39952200bd7152f809c3037c157b6978e387028768f99e89e39668402d4aa634" Jul 06 23:50:56.411861 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:56.411827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-prg5s" event={"ID":"1c1afb1b-4e1e-47c9-9279-35f4e103d5a7","Type":"ContainerStarted","Data":"74a0221987a6dd7c5070788b00dc7e1b718ed11f6b1ab2cc53ad2ff741ba807b"} Jul 06 23:50:58.293671 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:58.293637 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:50:58.294030 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:50:58.293720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:51:05.451244 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:05.451198 2564 generic.go:356] "Generic (PLEG): container finished" podID="bdd76a4e-600e-458e-adb3-801d5f30c4be" containerID="c70eff363cd2c1e4ac6bf9d519dbf925e52f9ee0c664987c81bc176ece65a940" exitCode=0 Jul 06 23:51:05.451650 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:05.451270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" event={"ID":"bdd76a4e-600e-458e-adb3-801d5f30c4be","Type":"ContainerDied","Data":"c70eff363cd2c1e4ac6bf9d519dbf925e52f9ee0c664987c81bc176ece65a940"} Jul 06 23:51:05.451650 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:05.451619 2564 scope.go:117] "RemoveContainer" containerID="c70eff363cd2c1e4ac6bf9d519dbf925e52f9ee0c664987c81bc176ece65a940" Jul 06 23:51:06.456687 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:06.456652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xgj9q" event={"ID":"bdd76a4e-600e-458e-adb3-801d5f30c4be","Type":"ContainerStarted","Data":"16fb4e157401b87ecd785403bf7b0da83a7fa433f71a3ed1bb79026147e58659"} Jul 06 23:51:09.471511 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:09.471466 2564 generic.go:356] "Generic (PLEG): container finished" podID="8e28fdd5-ea71-4ec9-a304-4dff04bf11f3" containerID="e44d963479d8fd2edd315afa8f3d4f5f95dd9f7f83caa2a771016c184149c675" exitCode=0 Jul 06 23:51:09.471783 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:09.471536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" event={"ID":"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3","Type":"ContainerDied","Data":"e44d963479d8fd2edd315afa8f3d4f5f95dd9f7f83caa2a771016c184149c675"} Jul 06 23:51:09.471931 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:09.471915 2564 scope.go:117] "RemoveContainer" containerID="e44d963479d8fd2edd315afa8f3d4f5f95dd9f7f83caa2a771016c184149c675" Jul 06 23:51:10.477485 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:10.477446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-hbn4z" event={"ID":"8e28fdd5-ea71-4ec9-a304-4dff04bf11f3","Type":"ContainerStarted","Data":"4aeb0e37639e8447120c005609423cb770de2f7ca302f0e2b9517e7e10ac6e0d"} Jul 06 23:51:18.300395 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:18.300364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:51:18.306278 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:18.306243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bd45d5848-ssflm" Jul 06 23:51:20.350406 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.350369 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-64c65fbd4c-r7kvf" podUID="f2c8c166-0b43-4994-ae7b-dad6f83a91e1" containerName="console" containerID="cri-o://1358b96862f3fd62e89f891f0e91dde2626cb32ce68b4766c4683a57e372c808" gracePeriod=15 Jul 06 23:51:20.520686 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.520654 2564 generic.go:356] "Generic (PLEG): container finished" podID="f2c8c166-0b43-4994-ae7b-dad6f83a91e1" containerID="1358b96862f3fd62e89f891f0e91dde2626cb32ce68b4766c4683a57e372c808" exitCode=0 Jul 06 23:51:20.520860 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.520704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c65fbd4c-r7kvf" event={"ID":"f2c8c166-0b43-4994-ae7b-dad6f83a91e1","Type":"ContainerDied","Data":"1358b96862f3fd62e89f891f0e91dde2626cb32ce68b4766c4683a57e372c808"} Jul 06 23:51:20.607664 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.607607 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:51:20.695040 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.694807 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-trusted-ca-bundle\") pod \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " Jul 06 23:51:20.695298 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.695280 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-config\") pod \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " Jul 06 23:51:20.695414 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.695358 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f2c8c166-0b43-4994-ae7b-dad6f83a91e1" (UID: "f2c8c166-0b43-4994-ae7b-dad6f83a91e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:20.695592 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.695441 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbkzq\" (UniqueName: \"kubernetes.io/projected/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-kube-api-access-qbkzq\") pod \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " Jul 06 23:51:20.695592 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.695484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-oauth-config\") pod \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " Jul 06 23:51:20.695704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.695592 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-serving-cert\") pod \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " Jul 06 23:51:20.695704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.695640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-oauth-serving-cert\") pod \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " Jul 06 23:51:20.695704 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.695676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-service-ca\") pod \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\" (UID: \"f2c8c166-0b43-4994-ae7b-dad6f83a91e1\") " Jul 06 23:51:20.696190 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.696043 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:20.697064 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.696817 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-config" (OuterVolumeSpecName: "console-config") pod "f2c8c166-0b43-4994-ae7b-dad6f83a91e1" (UID: "f2c8c166-0b43-4994-ae7b-dad6f83a91e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:20.699054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.699027 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f2c8c166-0b43-4994-ae7b-dad6f83a91e1" (UID: "f2c8c166-0b43-4994-ae7b-dad6f83a91e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:20.699350 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.699330 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f2c8c166-0b43-4994-ae7b-dad6f83a91e1" (UID: "f2c8c166-0b43-4994-ae7b-dad6f83a91e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:20.699566 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.699545 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "f2c8c166-0b43-4994-ae7b-dad6f83a91e1" (UID: "f2c8c166-0b43-4994-ae7b-dad6f83a91e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:20.701374 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.701291 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-kube-api-access-qbkzq" (OuterVolumeSpecName: "kube-api-access-qbkzq") pod "f2c8c166-0b43-4994-ae7b-dad6f83a91e1" (UID: "f2c8c166-0b43-4994-ae7b-dad6f83a91e1"). InnerVolumeSpecName "kube-api-access-qbkzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:20.701886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.701861 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f2c8c166-0b43-4994-ae7b-dad6f83a91e1" (UID: "f2c8c166-0b43-4994-ae7b-dad6f83a91e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:20.796998 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.796961 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:20.796998 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.796991 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbkzq\" (UniqueName: \"kubernetes.io/projected/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-kube-api-access-qbkzq\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:20.796998 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.797001 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:20.797216 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.797011 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:20.797216 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.797019 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:20.797216 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:20.797027 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c8c166-0b43-4994-ae7b-dad6f83a91e1-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:21.525913 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:21.525879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c65fbd4c-r7kvf" event={"ID":"f2c8c166-0b43-4994-ae7b-dad6f83a91e1","Type":"ContainerDied","Data":"a36df6272aabab0e1244e6a26919d29a1bfca43bf9cfde74a16d3298710bddfe"} Jul 06 23:51:21.526438 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:21.525938 2564 scope.go:117] "RemoveContainer" containerID="1358b96862f3fd62e89f891f0e91dde2626cb32ce68b4766c4683a57e372c808" Jul 06 23:51:21.526438 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:21.525950 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c65fbd4c-r7kvf" Jul 06 23:51:21.547228 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:21.547194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c65fbd4c-r7kvf"] Jul 06 23:51:21.548852 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:21.548834 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64c65fbd4c-r7kvf"] Jul 06 23:51:21.804133 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:21.804048 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2c8c166-0b43-4994-ae7b-dad6f83a91e1" path="/var/lib/kubelet/pods/f2c8c166-0b43-4994-ae7b-dad6f83a91e1/volumes" Jul 06 23:51:40.317828 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:40.317791 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:40.344487 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:40.344456 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:40.615631 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:40.615563 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:58.312386 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.312357 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:51:58.312886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.312640 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="prometheus" containerID="cri-o://4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" gracePeriod=600 Jul 06 23:51:58.312886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.312699 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-web" containerID="cri-o://eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" gracePeriod=600 Jul 06 23:51:58.312886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.312728 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="config-reloader" containerID="cri-o://6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" gracePeriod=600 Jul 06 23:51:58.312886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.312753 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" gracePeriod=600 Jul 06 23:51:58.312886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.312866 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="thanos-sidecar" containerID="cri-o://6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" gracePeriod=600 Jul 06 23:51:58.313170 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.312851 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy" containerID="cri-o://e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" gracePeriod=600 Jul 06 23:51:58.559817 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.559792 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:58.658679 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.658650 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dc8ff5d88-msr8v"] Jul 06 23:51:58.659045 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659026 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659047 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659068 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-web" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659077 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-web" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659090 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c8c166-0b43-4994-ae7b-dad6f83a91e1" containerName="console" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659099 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c8c166-0b43-4994-ae7b-dad6f83a91e1" containerName="console" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659107 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="prometheus" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659115 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="prometheus" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659125 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="thanos-sidecar" Jul 06 23:51:58.659136 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659133 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="thanos-sidecar" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659149 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-thanos" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659157 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-thanos" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659168 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="init-config-reloader" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659176 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="init-config-reloader" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659190 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="config-reloader" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659219 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="config-reloader" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659296 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2c8c166-0b43-4994-ae7b-dad6f83a91e1" containerName="console" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659309 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="prometheus" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659318 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="thanos-sidecar" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659325 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659337 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-web" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659344 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="config-reloader" Jul 06 23:51:58.659568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.659353 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerName="kube-rbac-proxy-thanos" Jul 06 23:51:58.662623 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.662604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.664760 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.664733 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 23:51:58.664866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.664795 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 23:51:58.664866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.664834 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 23:51:58.665018 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.664998 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 23:51:58.665135 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.665099 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 23:51:58.665454 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.665435 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 23:51:58.665555 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.665438 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m6b92\"" Jul 06 23:51:58.665555 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.665442 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 23:51:58.672442 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.672421 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 23:51:58.674750 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.674731 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc8ff5d88-msr8v"] Jul 06 23:51:58.677651 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677628 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" exitCode=0 Jul 06 23:51:58.677651 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677651 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" exitCode=0 Jul 06 23:51:58.677779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677661 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" exitCode=0 Jul 06 23:51:58.677779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677669 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" exitCode=0 Jul 06 23:51:58.677779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677676 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" exitCode=0 Jul 06 23:51:58.677779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677683 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" exitCode=0 Jul 06 23:51:58.677779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677704 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:58.677779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} Jul 06 23:51:58.677779 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} Jul 06 23:51:58.678076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} Jul 06 23:51:58.678076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} Jul 06 23:51:58.678076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677846 2564 scope.go:117] "RemoveContainer" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.678076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} Jul 06 23:51:58.678076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} Jul 06 23:51:58.678076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.677920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb217e32-0f36-4d1c-a440-a9df49ffcfd8","Type":"ContainerDied","Data":"58cfbc12e8bbf695cdb053e01a62490a8600d3880b897c971c382b8c053b5b54"} Jul 06 23:51:58.686314 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.686301 2564 scope.go:117] "RemoveContainer" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.695290 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.695269 2564 scope.go:117] "RemoveContainer" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.702310 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.702292 2564 scope.go:117] "RemoveContainer" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.709437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709409 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-metrics-client-ca\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709575 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709575 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-thanos-prometheus-http-client-file\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709575 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709559 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-2\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709748 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709599 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-trusted-ca-bundle\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709748 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709624 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-serving-certs-ca-bundle\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709748 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709650 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709748 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709695 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-kube-rbac-proxy\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709748 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709725 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-metrics-client-certs\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709752 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-db\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709804 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-1\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709830 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-web-config\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709851 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709891 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-kubelet-serving-ca-bundle\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709927 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-tls\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709955 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-0\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.709986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.709986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config-out\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.710353 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.710020 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l879x\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-kube-api-access-l879x\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.710353 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.710057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-grpc-tls\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.710353 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.710085 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-tls-assets\") pod \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\" (UID: \"eb217e32-0f36-4d1c-a440-a9df49ffcfd8\") " Jul 06 23:51:58.711800 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.711771 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:58.712256 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.712230 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:58.712351 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.712333 2564 scope.go:117] "RemoveContainer" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.713278 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.713249 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:58.713365 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.713317 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:58.714089 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.714064 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:58.714463 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.714435 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:58.716003 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.715980 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.716451 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.716340 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:58.716690 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.716600 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:58.716945 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.716868 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:58.717737 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.717694 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config-out" (OuterVolumeSpecName: "config-out") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:58.718300 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.718271 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.718401 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.718304 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.720019 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.719981 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.720123 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.720100 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config" (OuterVolumeSpecName: "config") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.720428 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.720402 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.720701 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.720678 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.721248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.721222 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.721785 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.721766 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-kube-api-access-l879x" (OuterVolumeSpecName: "kube-api-access-l879x") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "kube-api-access-l879x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:58.724712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.724698 2564 scope.go:117] "RemoveContainer" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.730840 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.730818 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-web-config" (OuterVolumeSpecName: "web-config") pod "eb217e32-0f36-4d1c-a440-a9df49ffcfd8" (UID: "eb217e32-0f36-4d1c-a440-a9df49ffcfd8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:51:58.732567 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.732554 2564 scope.go:117] "RemoveContainer" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.739347 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.739333 2564 scope.go:117] "RemoveContainer" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.739585 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:51:58.739565 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": container with ID starting with 437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43 not found: ID does not exist" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.739665 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.739590 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} err="failed to get container status \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": rpc error: code = NotFound desc = could not find container \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": container with ID starting with 437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43 not found: ID does not exist" Jul 06 23:51:58.739665 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.739607 2564 scope.go:117] "RemoveContainer" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.739840 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:51:58.739827 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": container with ID starting with e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b not found: ID does not exist" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.739882 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.739844 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} err="failed to get container status \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": rpc error: code = NotFound desc = could not find container \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": container with ID starting with e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b not found: ID does not exist" Jul 06 23:51:58.739882 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.739857 2564 scope.go:117] "RemoveContainer" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.740065 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:51:58.740048 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": container with ID starting with eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627 not found: ID does not exist" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.740106 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740069 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} err="failed to get container status \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": rpc error: code = NotFound desc = could not find container \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": container with ID starting with eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627 not found: ID does not exist" Jul 06 23:51:58.740106 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740082 2564 scope.go:117] "RemoveContainer" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.740313 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:51:58.740296 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": container with ID starting with 6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2 not found: ID does not exist" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.740354 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740323 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} err="failed to get container status \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": rpc error: code = NotFound desc = could not find container \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": container with ID starting with 6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2 not found: ID does not exist" Jul 06 23:51:58.740354 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740340 2564 scope.go:117] "RemoveContainer" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.740618 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:51:58.740601 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": container with ID starting with 6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0 not found: ID does not exist" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.740671 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740622 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} err="failed to get container status \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": rpc error: code = NotFound desc = could not find container \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": container with ID starting with 6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0 not found: ID does not exist" Jul 06 23:51:58.740671 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740635 2564 scope.go:117] "RemoveContainer" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.740855 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:51:58.740841 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": container with ID starting with 4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c not found: ID does not exist" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.740893 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740858 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} err="failed to get container status \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": rpc error: code = NotFound desc = could not find container \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": container with ID starting with 4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c not found: ID does not exist" Jul 06 23:51:58.740893 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.740870 2564 scope.go:117] "RemoveContainer" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.741072 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:51:58.741058 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": container with ID starting with 8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13 not found: ID does not exist" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.741111 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741076 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13"} err="failed to get container status \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": rpc error: code = NotFound desc = could not find container \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": container with ID starting with 8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13 not found: ID does not exist" Jul 06 23:51:58.741111 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741088 2564 scope.go:117] "RemoveContainer" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.741306 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741288 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} err="failed to get container status \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": rpc error: code = NotFound desc = could not find container \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": container with ID starting with 437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43 not found: ID does not exist" Jul 06 23:51:58.741306 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741305 2564 scope.go:117] "RemoveContainer" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.741517 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741486 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} err="failed to get container status \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": rpc error: code = NotFound desc = could not find container \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": container with ID starting with e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b not found: ID does not exist" Jul 06 23:51:58.741570 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741518 2564 scope.go:117] "RemoveContainer" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.741738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741723 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} err="failed to get container status \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": rpc error: code = NotFound desc = could not find container \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": container with ID starting with eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627 not found: ID does not exist" Jul 06 23:51:58.741787 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741738 2564 scope.go:117] "RemoveContainer" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.741928 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741913 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} err="failed to get container status \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": rpc error: code = NotFound desc = could not find container \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": container with ID starting with 6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2 not found: ID does not exist" Jul 06 23:51:58.741964 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.741928 2564 scope.go:117] "RemoveContainer" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.742135 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742118 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} err="failed to get container status \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": rpc error: code = NotFound desc = could not find container \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": container with ID starting with 6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0 not found: ID does not exist" Jul 06 23:51:58.742188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742135 2564 scope.go:117] "RemoveContainer" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.742324 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742309 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} err="failed to get container status \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": rpc error: code = NotFound desc = could not find container \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": container with ID starting with 4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c not found: ID does not exist" Jul 06 23:51:58.742363 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742324 2564 scope.go:117] "RemoveContainer" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.742572 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742545 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13"} err="failed to get container status \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": rpc error: code = NotFound desc = could not find container \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": container with ID starting with 8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13 not found: ID does not exist" Jul 06 23:51:58.742626 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742576 2564 scope.go:117] "RemoveContainer" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.742791 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742775 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} err="failed to get container status \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": rpc error: code = NotFound desc = could not find container \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": container with ID starting with 437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43 not found: ID does not exist" Jul 06 23:51:58.742838 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.742793 2564 scope.go:117] "RemoveContainer" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.743023 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743007 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} err="failed to get container status \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": rpc error: code = NotFound desc = could not find container \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": container with ID starting with e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b not found: ID does not exist" Jul 06 23:51:58.743065 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743023 2564 scope.go:117] "RemoveContainer" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.743230 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743214 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} err="failed to get container status \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": rpc error: code = NotFound desc = could not find container \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": container with ID starting with eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627 not found: ID does not exist" Jul 06 23:51:58.743230 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743230 2564 scope.go:117] "RemoveContainer" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.743424 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743409 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} err="failed to get container status \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": rpc error: code = NotFound desc = could not find container \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": container with ID starting with 6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2 not found: ID does not exist" Jul 06 23:51:58.743531 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743424 2564 scope.go:117] "RemoveContainer" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.743652 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743633 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} err="failed to get container status \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": rpc error: code = NotFound desc = could not find container \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": container with ID starting with 6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0 not found: ID does not exist" Jul 06 23:51:58.743695 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743653 2564 scope.go:117] "RemoveContainer" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.743864 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743847 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} err="failed to get container status \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": rpc error: code = NotFound desc = could not find container \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": container with ID starting with 4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c not found: ID does not exist" Jul 06 23:51:58.743908 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.743864 2564 scope.go:117] "RemoveContainer" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.744080 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744065 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13"} err="failed to get container status \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": rpc error: code = NotFound desc = could not find container \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": container with ID starting with 8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13 not found: ID does not exist" Jul 06 23:51:58.744121 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744081 2564 scope.go:117] "RemoveContainer" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.744278 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744261 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} err="failed to get container status \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": rpc error: code = NotFound desc = could not find container \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": container with ID starting with 437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43 not found: ID does not exist" Jul 06 23:51:58.744278 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744277 2564 scope.go:117] "RemoveContainer" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.744438 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744423 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} err="failed to get container status \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": rpc error: code = NotFound desc = could not find container \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": container with ID starting with e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b not found: ID does not exist" Jul 06 23:51:58.744476 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744437 2564 scope.go:117] "RemoveContainer" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.744635 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744621 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} err="failed to get container status \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": rpc error: code = NotFound desc = could not find container \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": container with ID starting with eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627 not found: ID does not exist" Jul 06 23:51:58.744677 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744635 2564 scope.go:117] "RemoveContainer" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.744834 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744818 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} err="failed to get container status \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": rpc error: code = NotFound desc = could not find container \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": container with ID starting with 6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2 not found: ID does not exist" Jul 06 23:51:58.744878 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.744836 2564 scope.go:117] "RemoveContainer" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.745040 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745026 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} err="failed to get container status \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": rpc error: code = NotFound desc = could not find container \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": container with ID starting with 6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0 not found: ID does not exist" Jul 06 23:51:58.745081 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745040 2564 scope.go:117] "RemoveContainer" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.745212 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745197 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} err="failed to get container status \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": rpc error: code = NotFound desc = could not find container \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": container with ID starting with 4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c not found: ID does not exist" Jul 06 23:51:58.745251 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745215 2564 scope.go:117] "RemoveContainer" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.745399 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745385 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13"} err="failed to get container status \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": rpc error: code = NotFound desc = could not find container \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": container with ID starting with 8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13 not found: ID does not exist" Jul 06 23:51:58.745438 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745399 2564 scope.go:117] "RemoveContainer" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.745637 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745621 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} err="failed to get container status \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": rpc error: code = NotFound desc = could not find container \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": container with ID starting with 437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43 not found: ID does not exist" Jul 06 23:51:58.745691 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745637 2564 scope.go:117] "RemoveContainer" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.745825 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745803 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} err="failed to get container status \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": rpc error: code = NotFound desc = could not find container \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": container with ID starting with e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b not found: ID does not exist" Jul 06 23:51:58.745866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745826 2564 scope.go:117] "RemoveContainer" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.746002 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.745988 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} err="failed to get container status \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": rpc error: code = NotFound desc = could not find container \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": container with ID starting with eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627 not found: ID does not exist" Jul 06 23:51:58.746036 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746001 2564 scope.go:117] "RemoveContainer" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.746187 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746170 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} err="failed to get container status \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": rpc error: code = NotFound desc = could not find container \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": container with ID starting with 6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2 not found: ID does not exist" Jul 06 23:51:58.746233 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746189 2564 scope.go:117] "RemoveContainer" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.746361 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746344 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} err="failed to get container status \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": rpc error: code = NotFound desc = could not find container \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": container with ID starting with 6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0 not found: ID does not exist" Jul 06 23:51:58.746403 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746363 2564 scope.go:117] "RemoveContainer" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.746649 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746630 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} err="failed to get container status \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": rpc error: code = NotFound desc = could not find container \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": container with ID starting with 4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c not found: ID does not exist" Jul 06 23:51:58.746694 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746649 2564 scope.go:117] "RemoveContainer" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.746885 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746868 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13"} err="failed to get container status \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": rpc error: code = NotFound desc = could not find container \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": container with ID starting with 8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13 not found: ID does not exist" Jul 06 23:51:58.746961 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.746886 2564 scope.go:117] "RemoveContainer" containerID="437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43" Jul 06 23:51:58.747118 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747096 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43"} err="failed to get container status \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": rpc error: code = NotFound desc = could not find container \"437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43\": container with ID starting with 437dfea04d176c9bb680811e386d710fb8657db8382ae395c9492d28680ebc43 not found: ID does not exist" Jul 06 23:51:58.747165 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747119 2564 scope.go:117] "RemoveContainer" containerID="e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b" Jul 06 23:51:58.747349 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747333 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b"} err="failed to get container status \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": rpc error: code = NotFound desc = could not find container \"e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b\": container with ID starting with e0185acb0927efd1fdcb6718e46f75116be02ef0cf08fab784abd9dc4244148b not found: ID does not exist" Jul 06 23:51:58.747391 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747351 2564 scope.go:117] "RemoveContainer" containerID="eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627" Jul 06 23:51:58.747608 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747590 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627"} err="failed to get container status \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": rpc error: code = NotFound desc = could not find container \"eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627\": container with ID starting with eb5f274587c682a38d4ef4dcdd1751a9861704c6cbff967b022f861ad4485627 not found: ID does not exist" Jul 06 23:51:58.747661 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747608 2564 scope.go:117] "RemoveContainer" containerID="6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2" Jul 06 23:51:58.747814 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747799 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2"} err="failed to get container status \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": rpc error: code = NotFound desc = could not find container \"6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2\": container with ID starting with 6cd26bf74d8cefbdac8ec2748d325585ae95d8d726d41df8765f42a926eb75e2 not found: ID does not exist" Jul 06 23:51:58.747853 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747813 2564 scope.go:117] "RemoveContainer" containerID="6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0" Jul 06 23:51:58.748014 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.747999 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0"} err="failed to get container status \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": rpc error: code = NotFound desc = could not find container \"6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0\": container with ID starting with 6b3deb6bb0e438875ab7a498e6df984076f926cc42913e3a3286569314632af0 not found: ID does not exist" Jul 06 23:51:58.748057 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.748015 2564 scope.go:117] "RemoveContainer" containerID="4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c" Jul 06 23:51:58.748229 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.748214 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c"} err="failed to get container status \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": rpc error: code = NotFound desc = could not find container \"4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c\": container with ID starting with 4bccd7eab27e6022ceca98a6fbef7f42a3a13a4c86ee2fd30f161f2f11af7d5c not found: ID does not exist" Jul 06 23:51:58.748229 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.748229 2564 scope.go:117] "RemoveContainer" containerID="8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13" Jul 06 23:51:58.748447 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.748433 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13"} err="failed to get container status \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": rpc error: code = NotFound desc = could not find container \"8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13\": container with ID starting with 8c032cae8a709723fa1373f901206e2c7a308b4cb8b9d6465e968ad96f524a13 not found: ID does not exist" Jul 06 23:51:58.811137 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-serving-cert\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.811254 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-trusted-ca-bundle\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.811254 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-console-config\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.811254 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-oauth-serving-cert\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.811345 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-service-ca\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.811345 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-oauth-config\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.811407 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/04db2e9e-608c-4af4-858c-4d1191e59b87-kube-api-access-7df54\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.811436 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811413 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811436 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811425 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-web-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811436 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811434 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811444 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811453 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811461 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811470 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-config-out\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811477 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l879x\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-kube-api-access-l879x\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811485 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-grpc-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811510 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-tls-assets\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811522 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-metrics-client-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811530 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811539 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811548 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811559 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811556 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811565 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811573 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811582 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-kube-rbac-proxy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811591 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-secret-metrics-client-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.811877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.811599 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb217e32-0f36-4d1c-a440-a9df49ffcfd8-prometheus-k8s-db\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:51:58.912902 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.912821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-trusted-ca-bundle\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.912902 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.912866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-console-config\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.912902 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.912882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-oauth-serving-cert\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.912902 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.912899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-service-ca\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.913193 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.912950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-oauth-config\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.913193 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.912975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/04db2e9e-608c-4af4-858c-4d1191e59b87-kube-api-access-7df54\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.913193 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.913016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-serving-cert\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.913742 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.913666 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-console-config\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.913742 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.913699 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-oauth-serving-cert\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.913742 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.913731 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-service-ca\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.913742 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.913800 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-trusted-ca-bundle\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.915475 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.915448 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-oauth-config\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.915588 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.915564 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-serving-cert\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.920026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.920009 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/04db2e9e-608c-4af4-858c-4d1191e59b87-kube-api-access-7df54\") pod \"console-6dc8ff5d88-msr8v\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:58.973653 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:58.973629 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:51:59.004340 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.004307 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:51:59.008629 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.008606 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:51:59.043771 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.043722 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:51:59.063054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.062996 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:51:59.063192 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.063159 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.065619 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.065620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.065790 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.065801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.066279 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.066414 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.066543 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.066565 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9g4betd0p8g1n\"" Jul 06 23:51:59.066738 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.066661 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fkcqz\"" Jul 06 23:51:59.067321 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.067090 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 23:51:59.067321 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.067100 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 23:51:59.067321 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.067209 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 23:51:59.067321 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.067318 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 23:51:59.067569 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.067354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 23:51:59.070058 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.069994 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 23:51:59.071786 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.071766 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 23:51:59.128035 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.128005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc8ff5d88-msr8v"] Jul 06 23:51:59.130876 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:51:59.130849 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04db2e9e_608c_4af4_858c_4d1191e59b87.slice/crio-164def0aa676d4e1e9d20c6b6978a48ce391de1fad81dbea44209351b5e6f359 WatchSource:0}: Error finding container 164def0aa676d4e1e9d20c6b6978a48ce391de1fad81dbea44209351b5e6f359: Status 404 returned error can't find the container with id 164def0aa676d4e1e9d20c6b6978a48ce391de1fad81dbea44209351b5e6f359 Jul 06 23:51:59.215687 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.215809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa98467-de04-4a8f-87d8-989121144d81-config-out\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.215809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.215877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.215877 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.215940 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa98467-de04-4a8f-87d8-989121144d81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.215940 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215929 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216008 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.215950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfng\" (UniqueName: \"kubernetes.io/projected/2aa98467-de04-4a8f-87d8-989121144d81-kube-api-access-blfng\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216152 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216152 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216107 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-config\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216152 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216234 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216234 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216234 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216234 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216348 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.216348 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.216259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-web-config\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.316862 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.316829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.316873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.316903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa98467-de04-4a8f-87d8-989121144d81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.316935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.316959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.316991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfng\" (UniqueName: \"kubernetes.io/projected/2aa98467-de04-4a8f-87d8-989121144d81-kube-api-access-blfng\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-config\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.317268 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-web-config\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa98467-de04-4a8f-87d8-989121144d81-config-out\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.317753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318330 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.318163 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318330 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.318208 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.318330 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.318236 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.319277 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.318947 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.319378 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.319332 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.320446 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.320417 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.320446 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.320439 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.320684 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.320662 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.320891 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.320870 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa98467-de04-4a8f-87d8-989121144d81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.321605 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.321481 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-web-config\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.321605 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.321562 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.321771 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.321734 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa98467-de04-4a8f-87d8-989121144d81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.322003 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.321974 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.322227 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.322209 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa98467-de04-4a8f-87d8-989121144d81-config-out\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.322832 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.322813 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.323073 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.323056 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.323248 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.323233 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-config\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.323629 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.323612 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa98467-de04-4a8f-87d8-989121144d81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.327761 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.327742 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfng\" (UniqueName: \"kubernetes.io/projected/2aa98467-de04-4a8f-87d8-989121144d81-kube-api-access-blfng\") pod \"prometheus-k8s-0\" (UID: \"2aa98467-de04-4a8f-87d8-989121144d81\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.376429 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.376400 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:51:59.520607 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.520571 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 23:51:59.524992 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:51:59.524963 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa98467_de04_4a8f_87d8_989121144d81.slice/crio-c58585011913b8d427adaa1665f0a14c0da24c674e212e87bfc97e0837b90b6a WatchSource:0}: Error finding container c58585011913b8d427adaa1665f0a14c0da24c674e212e87bfc97e0837b90b6a: Status 404 returned error can't find the container with id c58585011913b8d427adaa1665f0a14c0da24c674e212e87bfc97e0837b90b6a Jul 06 23:51:59.683546 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.683457 2564 generic.go:356] "Generic (PLEG): container finished" podID="2aa98467-de04-4a8f-87d8-989121144d81" containerID="90454545a2246f4f4c1caa3755b2543b8304cbe85f47c115ba4d6ebc405947fa" exitCode=0 Jul 06 23:51:59.683697 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.683550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerDied","Data":"90454545a2246f4f4c1caa3755b2543b8304cbe85f47c115ba4d6ebc405947fa"} Jul 06 23:51:59.683697 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.683634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerStarted","Data":"c58585011913b8d427adaa1665f0a14c0da24c674e212e87bfc97e0837b90b6a"} Jul 06 23:51:59.686203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.686180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8ff5d88-msr8v" event={"ID":"04db2e9e-608c-4af4-858c-4d1191e59b87","Type":"ContainerStarted","Data":"e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194"} Jul 06 23:51:59.686303 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.686218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8ff5d88-msr8v" event={"ID":"04db2e9e-608c-4af4-858c-4d1191e59b87","Type":"ContainerStarted","Data":"164def0aa676d4e1e9d20c6b6978a48ce391de1fad81dbea44209351b5e6f359"} Jul 06 23:51:59.724761 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.724719 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dc8ff5d88-msr8v" podStartSLOduration=1.7247037509999998 podStartE2EDuration="1.724703751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:51:59.72335612 +0000 UTC m=+184.373778091" watchObservedRunningTime="2026-07-06 23:51:59.724703751 +0000 UTC m=+184.375125698" Jul 06 23:51:59.807022 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:51:59.806760 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb217e32-0f36-4d1c-a440-a9df49ffcfd8" path="/var/lib/kubelet/pods/eb217e32-0f36-4d1c-a440-a9df49ffcfd8/volumes" Jul 06 23:52:00.694208 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:00.694170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerStarted","Data":"6d742c57ae4315394a55531bf2faae9aab0e25e08380b2e960df5130f4ab42d1"} Jul 06 23:52:00.694596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:00.694243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerStarted","Data":"655d66a3d48c54cb2ebb32e00d1b8ed88f537401bdc6e320c06ced23edd1e0d8"} Jul 06 23:52:00.694596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:00.694269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerStarted","Data":"2fe0a35f91a2a43c65d6131f7bfea06cfe6ecf1e1f21101070e5b47ddd1b44ba"} Jul 06 23:52:00.694596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:00.694294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerStarted","Data":"311278fc7b3392c6eee8e57f8c90d9e67cbe576035171e08096b3eb7b85adc1c"} Jul 06 23:52:00.694596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:00.694317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerStarted","Data":"27d6d2b73102e336b10bd6ca01b23c1e24220e7e620241265d0ec37469f013e0"} Jul 06 23:52:00.694596 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:00.694339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa98467-de04-4a8f-87d8-989121144d81","Type":"ContainerStarted","Data":"98117cc472cb3a6fc509022b1a3dbc18309d9b43abf2cf860f6424dfbdbf751a"} Jul 06 23:52:00.720813 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:00.720767 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.6993527259999999 podStartE2EDuration="1.720751511s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.398785ms" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:52:00.719236373 +0000 UTC m=+185.369658346" watchObservedRunningTime="2026-07-06 23:52:00.720751511 +0000 UTC m=+185.371173457" Jul 06 23:52:04.377113 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:04.377074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:52:08.973863 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:08.973829 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:52:08.974302 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:08.973886 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:52:08.978709 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:08.978684 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:52:09.735462 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:09.735437 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:52:59.377256 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:59.377220 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:52:59.392997 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:59.392971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:52:59.949825 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:52:59.949799 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 23:53:48.364556 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.364522 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx"] Jul 06 23:53:48.367679 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.367653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.369939 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.369918 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 23:53:48.370329 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.370311 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8sfwg\"" Jul 06 23:53:48.370644 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.370622 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 23:53:48.370833 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.370664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 23:53:48.370833 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.370706 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 23:53:48.387182 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.387153 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx"] Jul 06 23:53:48.416441 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.416415 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf10b018-5df1-4bda-82bd-64f925fbcf6e-tmp\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.416589 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.416458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf10b018-5df1-4bda-82bd-64f925fbcf6e-manifests\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.416589 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.416491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf10b018-5df1-4bda-82bd-64f925fbcf6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.416589 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.416583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf10b018-5df1-4bda-82bd-64f925fbcf6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.416723 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.416610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kff\" (UniqueName: \"kubernetes.io/projected/bf10b018-5df1-4bda-82bd-64f925fbcf6e-kube-api-access-68kff\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.516926 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.516892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf10b018-5df1-4bda-82bd-64f925fbcf6e-tmp\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.517076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.516936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf10b018-5df1-4bda-82bd-64f925fbcf6e-manifests\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.517076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.516962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf10b018-5df1-4bda-82bd-64f925fbcf6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.517076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.517012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf10b018-5df1-4bda-82bd-64f925fbcf6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.517076 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.517038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kff\" (UniqueName: \"kubernetes.io/projected/bf10b018-5df1-4bda-82bd-64f925fbcf6e-kube-api-access-68kff\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.517329 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.517305 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf10b018-5df1-4bda-82bd-64f925fbcf6e-manifests\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.519258 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.519230 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf10b018-5df1-4bda-82bd-64f925fbcf6e-tmp\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.519389 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.519369 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf10b018-5df1-4bda-82bd-64f925fbcf6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.519441 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.519432 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf10b018-5df1-4bda-82bd-64f925fbcf6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.527377 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.527356 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kff\" (UniqueName: \"kubernetes.io/projected/bf10b018-5df1-4bda-82bd-64f925fbcf6e-kube-api-access-68kff\") pod \"opendatahub-operator-controller-manager-64648fd55c-z7rpx\" (UID: \"bf10b018-5df1-4bda-82bd-64f925fbcf6e\") " pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.678646 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.678619 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:48.813014 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:48.812986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx"] Jul 06 23:53:48.816115 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:53:48.816084 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf10b018_5df1_4bda_82bd_64f925fbcf6e.slice/crio-6d96bda5da639e60bd90203dd9125df8ace360b79615ced68e40cb9a3977f701 WatchSource:0}: Error finding container 6d96bda5da639e60bd90203dd9125df8ace360b79615ced68e40cb9a3977f701: Status 404 returned error can't find the container with id 6d96bda5da639e60bd90203dd9125df8ace360b79615ced68e40cb9a3977f701 Jul 06 23:53:49.133755 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:49.133724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" event={"ID":"bf10b018-5df1-4bda-82bd-64f925fbcf6e","Type":"ContainerStarted","Data":"6d96bda5da639e60bd90203dd9125df8ace360b79615ced68e40cb9a3977f701"} Jul 06 23:53:52.151911 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:52.151874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" event={"ID":"bf10b018-5df1-4bda-82bd-64f925fbcf6e","Type":"ContainerStarted","Data":"32eca7ed0e5b058ec4b4acef1fd58150303c80818182858012370131abf17f32"} Jul 06 23:53:53.157243 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:53.157207 2564 generic.go:356] "Generic (PLEG): container finished" podID="bf10b018-5df1-4bda-82bd-64f925fbcf6e" containerID="32eca7ed0e5b058ec4b4acef1fd58150303c80818182858012370131abf17f32" exitCode=0 Jul 06 23:53:53.157634 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:53.157257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" event={"ID":"bf10b018-5df1-4bda-82bd-64f925fbcf6e","Type":"ContainerDied","Data":"32eca7ed0e5b058ec4b4acef1fd58150303c80818182858012370131abf17f32"} Jul 06 23:53:54.164746 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:54.164716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" event={"ID":"bf10b018-5df1-4bda-82bd-64f925fbcf6e","Type":"ContainerStarted","Data":"06830aaf9e7aa3c11b9fa02f04ffb4bac35cf8e87f065d324730c3b446874155"} Jul 06 23:53:54.165152 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:54.164966 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:53:54.185556 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:54.185488 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" podStartSLOduration=2.02823058 podStartE2EDuration="6.185473339s" totalImagesPullingTime="3.101522884s" totalInitContainerRuntime="1.055719875s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:53:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:53:54.183315914 +0000 UTC m=+298.833737872" watchObservedRunningTime="2026-07-06 23:53:54.185473339 +0000 UTC m=+298.835895337" Jul 06 23:53:55.684389 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:55.684364 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 06 23:53:55.688705 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:55.688679 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 06 23:53:55.697184 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:53:55.697165 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 23:54:05.172421 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:05.172392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64648fd55c-z7rpx" Jul 06 23:54:14.709326 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.709288 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh"] Jul 06 23:54:14.713768 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.713747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.716421 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.716402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-q8bb6\"" Jul 06 23:54:14.716751 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.716737 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 23:54:14.738676 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.738643 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh"] Jul 06 23:54:14.839688 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.839880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.839880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.839880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.839880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfzc\" (UniqueName: \"kubernetes.io/projected/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-kube-api-access-zlfzc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.839880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.840094 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.840094 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.839971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.840094 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.840013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940475 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940649 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940649 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940649 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940649 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940871 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940871 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfzc\" (UniqueName: \"kubernetes.io/projected/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-kube-api-access-zlfzc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940871 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.940871 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.941074 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.940991 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.941074 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.941064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.941393 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.941370 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.941550 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.941430 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.941880 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.941604 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.944012 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.943987 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.944158 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.944138 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.949197 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.949171 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:14.949437 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:14.949420 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfzc\" (UniqueName: \"kubernetes.io/projected/fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8-kube-api-access-zlfzc\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4prhvh\" (UID: \"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:15.024687 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:15.024600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:15.170919 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:15.170880 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh"] Jul 06 23:54:15.174705 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:54:15.174664 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5d3f56_d9a0_4d0b_a112_6cd21e83c5c8.slice/crio-85e078b75dc374897cb512989f0c351b09e29f2eadad001f554a5e56c3b705f2 WatchSource:0}: Error finding container 85e078b75dc374897cb512989f0c351b09e29f2eadad001f554a5e56c3b705f2: Status 404 returned error can't find the container with id 85e078b75dc374897cb512989f0c351b09e29f2eadad001f554a5e56c3b705f2 Jul 06 23:54:15.176568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:15.176552 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:54:15.260482 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:15.260450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" event={"ID":"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8","Type":"ContainerStarted","Data":"85e078b75dc374897cb512989f0c351b09e29f2eadad001f554a5e56c3b705f2"} Jul 06 23:54:17.359849 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.359814 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx"] Jul 06 23:54:17.363408 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.363388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.370397 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.370376 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wkzmh\"" Jul 06 23:54:17.370528 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.370380 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:54:17.370528 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.370380 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 23:54:17.371054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.371034 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 23:54:17.371054 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.371047 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 23:54:17.371464 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.371445 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 23:54:17.393968 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.393944 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx"] Jul 06 23:54:17.466203 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.466161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56fc5e67-0301-4ad4-adeb-dd1beb086775-cert\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.466373 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.466245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brd2p\" (UniqueName: \"kubernetes.io/projected/56fc5e67-0301-4ad4-adeb-dd1beb086775-kube-api-access-brd2p\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.466373 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.466293 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56fc5e67-0301-4ad4-adeb-dd1beb086775-manager-config\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.466486 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.466375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56fc5e67-0301-4ad4-adeb-dd1beb086775-metrics-cert\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.567540 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.567469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56fc5e67-0301-4ad4-adeb-dd1beb086775-metrics-cert\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.567724 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.567599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56fc5e67-0301-4ad4-adeb-dd1beb086775-cert\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.567724 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.567683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brd2p\" (UniqueName: \"kubernetes.io/projected/56fc5e67-0301-4ad4-adeb-dd1beb086775-kube-api-access-brd2p\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.567848 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.567726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56fc5e67-0301-4ad4-adeb-dd1beb086775-manager-config\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.568678 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.568639 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56fc5e67-0301-4ad4-adeb-dd1beb086775-manager-config\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.570483 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.570447 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56fc5e67-0301-4ad4-adeb-dd1beb086775-cert\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.570615 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.570491 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56fc5e67-0301-4ad4-adeb-dd1beb086775-metrics-cert\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.580069 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.580041 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brd2p\" (UniqueName: \"kubernetes.io/projected/56fc5e67-0301-4ad4-adeb-dd1beb086775-kube-api-access-brd2p\") pod \"lws-controller-manager-f447c9d54-k7sjx\" (UID: \"56fc5e67-0301-4ad4-adeb-dd1beb086775\") " pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.675515 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.675462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:17.832968 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.832939 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw"] Jul 06 23:54:17.836744 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.836725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:17.839246 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.839223 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 23:54:17.839342 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.839307 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zlj6j\"" Jul 06 23:54:17.848193 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.848161 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9"] Jul 06 23:54:17.851991 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.851968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw"] Jul 06 23:54:17.852139 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.852072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:17.855165 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.855144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7r6dx\"" Jul 06 23:54:17.855266 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.855152 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 23:54:17.863955 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.863936 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9"] Jul 06 23:54:17.971243 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.971167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35e657e-8930-4d1a-952c-1ecc10f311a5-cert\") pod \"kserve-controller-manager-d6f4b65b8-lvpzw\" (UID: \"a35e657e-8930-4d1a-952c-1ecc10f311a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:17.971243 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.971219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmql\" (UniqueName: \"kubernetes.io/projected/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-kube-api-access-phmql\") pod \"llmisvc-controller-manager-777c6bd7d5-mx9r9\" (UID: \"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:17.971455 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.971330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpch\" (UniqueName: \"kubernetes.io/projected/a35e657e-8930-4d1a-952c-1ecc10f311a5-kube-api-access-xmpch\") pod \"kserve-controller-manager-d6f4b65b8-lvpzw\" (UID: \"a35e657e-8930-4d1a-952c-1ecc10f311a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:17.971455 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:17.971368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mx9r9\" (UID: \"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:18.071999 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.071969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmql\" (UniqueName: \"kubernetes.io/projected/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-kube-api-access-phmql\") pod \"llmisvc-controller-manager-777c6bd7d5-mx9r9\" (UID: \"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:18.072159 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.072041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpch\" (UniqueName: \"kubernetes.io/projected/a35e657e-8930-4d1a-952c-1ecc10f311a5-kube-api-access-xmpch\") pod \"kserve-controller-manager-d6f4b65b8-lvpzw\" (UID: \"a35e657e-8930-4d1a-952c-1ecc10f311a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:18.072159 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.072079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mx9r9\" (UID: \"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:18.072281 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.072168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35e657e-8930-4d1a-952c-1ecc10f311a5-cert\") pod \"kserve-controller-manager-d6f4b65b8-lvpzw\" (UID: \"a35e657e-8930-4d1a-952c-1ecc10f311a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:18.072281 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:54:18.072264 2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 23:54:18.072388 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:54:18.072329 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a35e657e-8930-4d1a-952c-1ecc10f311a5-cert podName:a35e657e-8930-4d1a-952c-1ecc10f311a5 nodeName:}" failed. No retries permitted until 2026-07-06 23:54:18.572309228 +0000 UTC m=+323.222731152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a35e657e-8930-4d1a-952c-1ecc10f311a5-cert") pod "kserve-controller-manager-d6f4b65b8-lvpzw" (UID: "a35e657e-8930-4d1a-952c-1ecc10f311a5") : secret "kserve-webhook-server-cert" not found Jul 06 23:54:18.072388 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:54:18.072363 2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 23:54:18.072520 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:54:18.072430 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-cert podName:d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c nodeName:}" failed. No retries permitted until 2026-07-06 23:54:18.57241267 +0000 UTC m=+323.222834611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-cert") pod "llmisvc-controller-manager-777c6bd7d5-mx9r9" (UID: "d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c") : secret "llmisvc-webhook-server-cert" not found Jul 06 23:54:18.126685 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.126635 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpch\" (UniqueName: \"kubernetes.io/projected/a35e657e-8930-4d1a-952c-1ecc10f311a5-kube-api-access-xmpch\") pod \"kserve-controller-manager-d6f4b65b8-lvpzw\" (UID: \"a35e657e-8930-4d1a-952c-1ecc10f311a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:18.127479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.127454 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmql\" (UniqueName: \"kubernetes.io/projected/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-kube-api-access-phmql\") pod \"llmisvc-controller-manager-777c6bd7d5-mx9r9\" (UID: \"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:18.578984 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.578947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35e657e-8930-4d1a-952c-1ecc10f311a5-cert\") pod \"kserve-controller-manager-d6f4b65b8-lvpzw\" (UID: \"a35e657e-8930-4d1a-952c-1ecc10f311a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:18.579304 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.579030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mx9r9\" (UID: \"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:18.581585 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.581560 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mx9r9\" (UID: \"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:18.581809 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.581791 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35e657e-8930-4d1a-952c-1ecc10f311a5-cert\") pod \"kserve-controller-manager-d6f4b65b8-lvpzw\" (UID: \"a35e657e-8930-4d1a-952c-1ecc10f311a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:18.617303 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.617244 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 06 23:54:18.617400 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.617363 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 06 23:54:18.617435 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.617407 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 06 23:54:18.686967 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.686857 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx"] Jul 06 23:54:18.689387 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:54:18.689362 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fc5e67_0301_4ad4_adeb_dd1beb086775.slice/crio-6787759df702372ef81a7a631ffbf89b6a3d69bdc75456e2be3bc5031c48c816 WatchSource:0}: Error finding container 6787759df702372ef81a7a631ffbf89b6a3d69bdc75456e2be3bc5031c48c816: Status 404 returned error can't find the container with id 6787759df702372ef81a7a631ffbf89b6a3d69bdc75456e2be3bc5031c48c816 Jul 06 23:54:18.750610 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.750583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:18.764153 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.764132 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:18.962339 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:18.962311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9"] Jul 06 23:54:18.965715 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:54:18.965689 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd3eb8e3c_a5cd_46ba_a641_9a339fc2e00c.slice/crio-a4547ee35b8e723911128f49b6f5fba3bf204130d1afdff6739f95eabedf889a WatchSource:0}: Error finding container a4547ee35b8e723911128f49b6f5fba3bf204130d1afdff6739f95eabedf889a: Status 404 returned error can't find the container with id a4547ee35b8e723911128f49b6f5fba3bf204130d1afdff6739f95eabedf889a Jul 06 23:54:19.133856 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:19.133826 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw"] Jul 06 23:54:19.137446 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:54:19.137398 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35e657e_8930_4d1a_952c_1ecc10f311a5.slice/crio-2048a5eb2fad8064c69f2b2d165e568ea6dd2badae65e4145dcf1e9f8f1fefde WatchSource:0}: Error finding container 2048a5eb2fad8064c69f2b2d165e568ea6dd2badae65e4145dcf1e9f8f1fefde: Status 404 returned error can't find the container with id 2048a5eb2fad8064c69f2b2d165e568ea6dd2badae65e4145dcf1e9f8f1fefde Jul 06 23:54:19.287371 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:19.287292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" event={"ID":"a35e657e-8930-4d1a-952c-1ecc10f311a5","Type":"ContainerStarted","Data":"2048a5eb2fad8064c69f2b2d165e568ea6dd2badae65e4145dcf1e9f8f1fefde"} Jul 06 23:54:19.290394 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:19.289991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" event={"ID":"fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8","Type":"ContainerStarted","Data":"af1998de166b3922456f853330cb47a2cd4073740c34521d6e35c199d0d6f8bc"} Jul 06 23:54:19.293628 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:19.293598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" event={"ID":"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c","Type":"ContainerStarted","Data":"a4547ee35b8e723911128f49b6f5fba3bf204130d1afdff6739f95eabedf889a"} Jul 06 23:54:19.295449 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:19.295408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" event={"ID":"56fc5e67-0301-4ad4-adeb-dd1beb086775","Type":"ContainerStarted","Data":"6787759df702372ef81a7a631ffbf89b6a3d69bdc75456e2be3bc5031c48c816"} Jul 06 23:54:19.320558 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:19.320098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" podStartSLOduration=1.8798423450000001 podStartE2EDuration="5.320078146s" totalImagesPullingTime="3.440235801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:54:19.317773823 +0000 UTC m=+323.968195796" watchObservedRunningTime="2026-07-06 23:54:19.320078146 +0000 UTC m=+323.970500096" Jul 06 23:54:20.025273 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:20.025124 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:20.032085 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:20.032052 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:20.305713 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:20.305604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:20.307411 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:20.307385 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4prhvh" Jul 06 23:54:23.321370 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.321333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" event={"ID":"d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c","Type":"ContainerStarted","Data":"53d1946bebc6bc85549e5c430704b83bbf4fda70e97ffd55bfbc61b7907b19c9"} Jul 06 23:54:23.321874 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.321527 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:23.322886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.322859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" event={"ID":"56fc5e67-0301-4ad4-adeb-dd1beb086775","Type":"ContainerStarted","Data":"502217c2ed4f85cc0871dc0dd2e7aca03e0fec21d25dcd2ef31cab155581c7ee"} Jul 06 23:54:23.323029 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.322975 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:23.324119 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.324101 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" event={"ID":"a35e657e-8930-4d1a-952c-1ecc10f311a5","Type":"ContainerStarted","Data":"5a0c2f9cf053ed07e73ba791561a521c4c725abe9d1a2eaca6ac369518add82a"} Jul 06 23:54:23.324245 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.324231 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:54:23.346198 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.346159 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" podStartSLOduration=2.610393973 podStartE2EDuration="6.346146454s" totalImagesPullingTime="3.735752481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:54:23.344508947 +0000 UTC m=+327.994930904" watchObservedRunningTime="2026-07-06 23:54:23.346146454 +0000 UTC m=+327.996568400" Jul 06 23:54:23.366667 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.366626 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" podStartSLOduration=2.8612522289999998 podStartE2EDuration="6.366612292s" totalImagesPullingTime="3.505360063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:54:23.364823609 +0000 UTC m=+328.015245576" watchObservedRunningTime="2026-07-06 23:54:23.366612292 +0000 UTC m=+328.017034220" Jul 06 23:54:23.382383 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:23.382342 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" podStartSLOduration=2.429226694 podStartE2EDuration="6.382329508s" totalImagesPullingTime="3.953102814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:54:23.382188952 +0000 UTC m=+328.032610909" watchObservedRunningTime="2026-07-06 23:54:23.382329508 +0000 UTC m=+328.032751457" Jul 06 23:54:34.330943 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:34.330915 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f447c9d54-k7sjx" Jul 06 23:54:54.331030 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:54.330946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mx9r9" Jul 06 23:54:54.334026 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:54:54.334005 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lvpzw" Jul 06 23:55:09.207655 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.207623 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r6v4"] Jul 06 23:55:09.218645 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.218625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:09.223595 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.223574 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 23:55:09.223775 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.223742 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-p7fnw\"" Jul 06 23:55:09.229188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.229171 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 23:55:09.232540 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.232519 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r6v4"] Jul 06 23:55:09.312221 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.312191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbp4v\" (UniqueName: \"kubernetes.io/projected/51eaaf9d-3481-4948-a048-5db2804d25f8-kube-api-access-lbp4v\") pod \"kuadrant-operator-catalog-9r6v4\" (UID: \"51eaaf9d-3481-4948-a048-5db2804d25f8\") " pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:09.412915 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.412875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbp4v\" (UniqueName: \"kubernetes.io/projected/51eaaf9d-3481-4948-a048-5db2804d25f8-kube-api-access-lbp4v\") pod \"kuadrant-operator-catalog-9r6v4\" (UID: \"51eaaf9d-3481-4948-a048-5db2804d25f8\") " pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:09.421331 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.421304 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbp4v\" (UniqueName: \"kubernetes.io/projected/51eaaf9d-3481-4948-a048-5db2804d25f8-kube-api-access-lbp4v\") pod \"kuadrant-operator-catalog-9r6v4\" (UID: \"51eaaf9d-3481-4948-a048-5db2804d25f8\") " pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:09.528329 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.528255 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:09.662936 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:09.662913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r6v4"] Jul 06 23:55:09.665128 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:55:09.665085 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51eaaf9d_3481_4948_a048_5db2804d25f8.slice/crio-73f0507c382338323aede5c764bda7b1c6a178c4f868f6ffb04b922378993ae5 WatchSource:0}: Error finding container 73f0507c382338323aede5c764bda7b1c6a178c4f868f6ffb04b922378993ae5: Status 404 returned error can't find the container with id 73f0507c382338323aede5c764bda7b1c6a178c4f868f6ffb04b922378993ae5 Jul 06 23:55:10.541954 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:10.541896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" event={"ID":"51eaaf9d-3481-4948-a048-5db2804d25f8","Type":"ContainerStarted","Data":"73f0507c382338323aede5c764bda7b1c6a178c4f868f6ffb04b922378993ae5"} Jul 06 23:55:12.552259 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:12.552223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" event={"ID":"51eaaf9d-3481-4948-a048-5db2804d25f8","Type":"ContainerStarted","Data":"66f3b64cc9216b3bd7685ce16469a6607dbaac839f85020b678b9371807441da"} Jul 06 23:55:12.566441 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:12.566398 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" podStartSLOduration=1.72528805 podStartE2EDuration="3.566383514s" totalImagesPullingTime="1.841095464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:55:12.565355577 +0000 UTC m=+377.215777546" watchObservedRunningTime="2026-07-06 23:55:12.566383514 +0000 UTC m=+377.216805461" Jul 06 23:55:19.529130 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:19.529098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:19.529557 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:19.529180 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:19.552445 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:19.552419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:19.604157 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:19.604127 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-9r6v4" Jul 06 23:55:36.002478 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.002447 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m"] Jul 06 23:55:36.006088 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.006073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" Jul 06 23:55:36.008786 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.008765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dmn8l\"" Jul 06 23:55:36.030928 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.030906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m"] Jul 06 23:55:36.149011 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.148976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmf4v\" (UniqueName: \"kubernetes.io/projected/1fc5f808-c2f8-4daa-9783-118e3e2fd530-kube-api-access-qmf4v\") pod \"authorino-operator-75cc8d5cb7-g2g5m\" (UID: \"1fc5f808-c2f8-4daa-9783-118e3e2fd530\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" Jul 06 23:55:36.249884 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.249848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmf4v\" (UniqueName: \"kubernetes.io/projected/1fc5f808-c2f8-4daa-9783-118e3e2fd530-kube-api-access-qmf4v\") pod \"authorino-operator-75cc8d5cb7-g2g5m\" (UID: \"1fc5f808-c2f8-4daa-9783-118e3e2fd530\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" Jul 06 23:55:36.262376 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.262326 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmf4v\" (UniqueName: \"kubernetes.io/projected/1fc5f808-c2f8-4daa-9783-118e3e2fd530-kube-api-access-qmf4v\") pod \"authorino-operator-75cc8d5cb7-g2g5m\" (UID: \"1fc5f808-c2f8-4daa-9783-118e3e2fd530\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" Jul 06 23:55:36.316182 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.316160 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" Jul 06 23:55:36.443386 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.443364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m"] Jul 06 23:55:36.445589 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:55:36.445559 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc5f808_c2f8_4daa_9783_118e3e2fd530.slice/crio-945e5edaa98b9b82c1a1b908630e311f39dae332a78fb624497e7b46ea54d300 WatchSource:0}: Error finding container 945e5edaa98b9b82c1a1b908630e311f39dae332a78fb624497e7b46ea54d300: Status 404 returned error can't find the container with id 945e5edaa98b9b82c1a1b908630e311f39dae332a78fb624497e7b46ea54d300 Jul 06 23:55:36.658265 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:36.658236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" event={"ID":"1fc5f808-c2f8-4daa-9783-118e3e2fd530","Type":"ContainerStarted","Data":"945e5edaa98b9b82c1a1b908630e311f39dae332a78fb624497e7b46ea54d300"} Jul 06 23:55:39.672839 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:39.672802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" event={"ID":"1fc5f808-c2f8-4daa-9783-118e3e2fd530","Type":"ContainerStarted","Data":"75bdb8ffcf34d44734b744bf14ceee2495f19e2c0652499bd86598c19321cb75"} Jul 06 23:55:39.673228 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:39.672946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" Jul 06 23:55:39.728519 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:39.728438 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" podStartSLOduration=2.16461418 podStartE2EDuration="4.728420481s" totalImagesPullingTime="2.563806301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:55:39.724043885 +0000 UTC m=+404.374465868" watchObservedRunningTime="2026-07-06 23:55:39.728420481 +0000 UTC m=+404.378842431" Jul 06 23:55:44.057360 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.057325 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6"] Jul 06 23:55:44.061728 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.061712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" Jul 06 23:55:44.063780 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.063758 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 23:55:44.063909 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.063802 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-svj2b\"" Jul 06 23:55:44.072618 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.072597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6"] Jul 06 23:55:44.108751 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.108718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxn6\" (UniqueName: \"kubernetes.io/projected/618e1dd4-2671-45c4-9f98-f67aefc3edda-kube-api-access-xmxn6\") pod \"dns-operator-controller-manager-864dd78c4c-sh4d6\" (UID: \"618e1dd4-2671-45c4-9f98-f67aefc3edda\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" Jul 06 23:55:44.209196 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.209166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxn6\" (UniqueName: \"kubernetes.io/projected/618e1dd4-2671-45c4-9f98-f67aefc3edda-kube-api-access-xmxn6\") pod \"dns-operator-controller-manager-864dd78c4c-sh4d6\" (UID: \"618e1dd4-2671-45c4-9f98-f67aefc3edda\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" Jul 06 23:55:44.222360 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.222328 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxn6\" (UniqueName: \"kubernetes.io/projected/618e1dd4-2671-45c4-9f98-f67aefc3edda-kube-api-access-xmxn6\") pod \"dns-operator-controller-manager-864dd78c4c-sh4d6\" (UID: \"618e1dd4-2671-45c4-9f98-f67aefc3edda\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" Jul 06 23:55:44.372127 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.372053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" Jul 06 23:55:44.515887 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.515840 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6"] Jul 06 23:55:44.519627 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:55:44.519601 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618e1dd4_2671_45c4_9f98_f67aefc3edda.slice/crio-f986288e31f9da236509b7c323844a1b5f99c0e8722a1307587ca78bb5bf2b9b WatchSource:0}: Error finding container f986288e31f9da236509b7c323844a1b5f99c0e8722a1307587ca78bb5bf2b9b: Status 404 returned error can't find the container with id f986288e31f9da236509b7c323844a1b5f99c0e8722a1307587ca78bb5bf2b9b Jul 06 23:55:44.704770 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:44.704735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" event={"ID":"618e1dd4-2671-45c4-9f98-f67aefc3edda","Type":"ContainerStarted","Data":"f986288e31f9da236509b7c323844a1b5f99c0e8722a1307587ca78bb5bf2b9b"} Jul 06 23:55:46.716645 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:46.716613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" event={"ID":"618e1dd4-2671-45c4-9f98-f67aefc3edda","Type":"ContainerStarted","Data":"570dcb6655e9b77db48ba1c584df2f6cf1c2c3e343ab51b6ac112a6e868ac0cb"} Jul 06 23:55:46.717083 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:46.716683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" Jul 06 23:55:46.745145 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:46.745100 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" podStartSLOduration=1.080157945 podStartE2EDuration="2.745086695s" totalImagesPullingTime="1.66492875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:55:46.743131091 +0000 UTC m=+411.393553039" watchObservedRunningTime="2026-07-06 23:55:46.745086695 +0000 UTC m=+411.395508640" Jul 06 23:55:49.603543 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.603479 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-555fdc4c8c-bwjvn"] Jul 06 23:55:49.607666 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.607637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.617415 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.617392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-555fdc4c8c-bwjvn"] Jul 06 23:55:49.651567 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.651541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-config\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.651712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.651577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-trusted-ca-bundle\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.651712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.651603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-oauth-config\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.651712 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.651618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-oauth-serving-cert\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.651836 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.651711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-service-ca\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.651836 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.651734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-serving-cert\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.651836 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.651749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6s68\" (UniqueName: \"kubernetes.io/projected/c7223f9b-03b6-46dd-9396-8fab3bbceffd-kube-api-access-q6s68\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.752846 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.752814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-trusted-ca-bundle\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.752854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-oauth-config\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.752875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-oauth-serving-cert\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.752918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-service-ca\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.752943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-serving-cert\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.752962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6s68\" (UniqueName: \"kubernetes.io/projected/c7223f9b-03b6-46dd-9396-8fab3bbceffd-kube-api-access-q6s68\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753020 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.753013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-config\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753751 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.753725 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-oauth-serving-cert\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753924 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.753900 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-service-ca\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.753999 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.753978 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-config\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.754055 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.754004 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7223f9b-03b6-46dd-9396-8fab3bbceffd-trusted-ca-bundle\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.755461 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.755434 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-serving-cert\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.755568 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.755478 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7223f9b-03b6-46dd-9396-8fab3bbceffd-console-oauth-config\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.766844 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.766825 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6s68\" (UniqueName: \"kubernetes.io/projected/c7223f9b-03b6-46dd-9396-8fab3bbceffd-kube-api-access-q6s68\") pod \"console-555fdc4c8c-bwjvn\" (UID: \"c7223f9b-03b6-46dd-9396-8fab3bbceffd\") " pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:49.919858 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:49.919820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:50.057040 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:50.057013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-555fdc4c8c-bwjvn"] Jul 06 23:55:50.060040 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:55:50.060013 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7223f9b_03b6_46dd_9396_8fab3bbceffd.slice/crio-69bc6481a23ca6b50cf250bff20dea006186c5e0e7b92c7a641925faddf91f0d WatchSource:0}: Error finding container 69bc6481a23ca6b50cf250bff20dea006186c5e0e7b92c7a641925faddf91f0d: Status 404 returned error can't find the container with id 69bc6481a23ca6b50cf250bff20dea006186c5e0e7b92c7a641925faddf91f0d Jul 06 23:55:50.679874 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:50.679830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g2g5m" Jul 06 23:55:50.738839 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:50.738806 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-555fdc4c8c-bwjvn" event={"ID":"c7223f9b-03b6-46dd-9396-8fab3bbceffd","Type":"ContainerStarted","Data":"962804a1df1a88ba1418d30b9040008bedbc0577d8b2549019aa3551c5cc0cd5"} Jul 06 23:55:50.739009 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:50.738855 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-555fdc4c8c-bwjvn" event={"ID":"c7223f9b-03b6-46dd-9396-8fab3bbceffd","Type":"ContainerStarted","Data":"69bc6481a23ca6b50cf250bff20dea006186c5e0e7b92c7a641925faddf91f0d"} Jul 06 23:55:50.760583 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:50.760489 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-555fdc4c8c-bwjvn" podStartSLOduration=1.760474604 podStartE2EDuration="1.760474604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:55:50.758086164 +0000 UTC m=+415.408508133" watchObservedRunningTime="2026-07-06 23:55:50.760474604 +0000 UTC m=+415.410896549" Jul 06 23:55:57.724578 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:57.724550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sh4d6" Jul 06 23:55:59.499652 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.499623 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc"] Jul 06 23:55:59.503339 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.503323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.505785 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.505768 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gk8sk\"" Jul 06 23:55:59.517081 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.517057 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc"] Jul 06 23:55:59.535857 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.535834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc757dda-ca43-4e5a-85bc-926468c9f266-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" (UID: \"dc757dda-ca43-4e5a-85bc-926468c9f266\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.535974 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.535880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmq6p\" (UniqueName: \"kubernetes.io/projected/dc757dda-ca43-4e5a-85bc-926468c9f266-kube-api-access-gmq6p\") pod \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" (UID: \"dc757dda-ca43-4e5a-85bc-926468c9f266\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.623560 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.623529 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc"] Jul 06 23:55:59.623902 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:55:59.623879 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-gmq6p], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" podUID="dc757dda-ca43-4e5a-85bc-926468c9f266" Jul 06 23:55:59.630551 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.630520 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc"] Jul 06 23:55:59.636564 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.636532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc757dda-ca43-4e5a-85bc-926468c9f266-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" (UID: \"dc757dda-ca43-4e5a-85bc-926468c9f266\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.636695 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.636614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmq6p\" (UniqueName: \"kubernetes.io/projected/dc757dda-ca43-4e5a-85bc-926468c9f266-kube-api-access-gmq6p\") pod \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" (UID: \"dc757dda-ca43-4e5a-85bc-926468c9f266\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.636946 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.636920 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc757dda-ca43-4e5a-85bc-926468c9f266-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" (UID: \"dc757dda-ca43-4e5a-85bc-926468c9f266\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.638730 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:55:59.638706 2564 projected.go:196] Error preparing data for projected volume kube-api-access-gmq6p for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-139-100.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object Jul 06 23:55:59.638838 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:55:59.638800 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc757dda-ca43-4e5a-85bc-926468c9f266-kube-api-access-gmq6p podName:dc757dda-ca43-4e5a-85bc-926468c9f266 nodeName:}" failed. No retries permitted until 2026-07-06 23:56:00.138777469 +0000 UTC m=+424.789199398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gmq6p" (UniqueName: "kubernetes.io/projected/dc757dda-ca43-4e5a-85bc-926468c9f266-kube-api-access-gmq6p") pod "kuadrant-operator-controller-manager-86878c769f-p2qmc" (UID: "dc757dda-ca43-4e5a-85bc-926468c9f266") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-139-100.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object Jul 06 23:55:59.648077 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.648050 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b"] Jul 06 23:55:59.652240 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.652219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:55:59.674637 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.674609 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b"] Jul 06 23:55:59.737585 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.737558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcd4\" (UniqueName: \"kubernetes.io/projected/8c22cde2-2de5-4760-9d35-8e396f71525c-kube-api-access-krcd4\") pod \"kuadrant-operator-controller-manager-86878c769f-ppg2b\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:55:59.737732 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.737670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c22cde2-2de5-4760-9d35-8e396f71525c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ppg2b\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:55:59.781303 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.781241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.783393 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:55:59.783368 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" podUID="dc757dda-ca43-4e5a-85bc-926468c9f266" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.787812 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.787794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.789568 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:55:59.789549 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" podUID="dc757dda-ca43-4e5a-85bc-926468c9f266" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:55:59.838668 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.838645 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc757dda-ca43-4e5a-85bc-926468c9f266-extensions-socket-volume\") pod \"dc757dda-ca43-4e5a-85bc-926468c9f266\" (UID: \"dc757dda-ca43-4e5a-85bc-926468c9f266\") " Jul 06 23:55:59.838788 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.838773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcd4\" (UniqueName: \"kubernetes.io/projected/8c22cde2-2de5-4760-9d35-8e396f71525c-kube-api-access-krcd4\") pod \"kuadrant-operator-controller-manager-86878c769f-ppg2b\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:55:59.838886 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.838865 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc757dda-ca43-4e5a-85bc-926468c9f266-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dc757dda-ca43-4e5a-85bc-926468c9f266" (UID: "dc757dda-ca43-4e5a-85bc-926468c9f266"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:55:59.838952 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.838870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c22cde2-2de5-4760-9d35-8e396f71525c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ppg2b\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:55:59.839059 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.839026 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmq6p\" (UniqueName: \"kubernetes.io/projected/dc757dda-ca43-4e5a-85bc-926468c9f266-kube-api-access-gmq6p\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:55:59.839059 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.839053 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc757dda-ca43-4e5a-85bc-926468c9f266-extensions-socket-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:55:59.839231 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.839177 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c22cde2-2de5-4760-9d35-8e396f71525c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ppg2b\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:55:59.849706 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.849683 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcd4\" (UniqueName: \"kubernetes.io/projected/8c22cde2-2de5-4760-9d35-8e396f71525c-kube-api-access-krcd4\") pod \"kuadrant-operator-controller-manager-86878c769f-ppg2b\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:55:59.920852 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.920823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:59.921030 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.920885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:59.925967 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.925947 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:55:59.963152 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:55:59.963116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:56:00.110986 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:00.110956 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b"] Jul 06 23:56:00.114179 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:56:00.114145 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c22cde2_2de5_4760_9d35_8e396f71525c.slice/crio-d3a435cf46ee2b1fa65d1d657372dbae4d71334222300c60c92667ef459b0c99 WatchSource:0}: Error finding container d3a435cf46ee2b1fa65d1d657372dbae4d71334222300c60c92667ef459b0c99: Status 404 returned error can't find the container with id d3a435cf46ee2b1fa65d1d657372dbae4d71334222300c60c92667ef459b0c99 Jul 06 23:56:00.788362 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:00.788326 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" event={"ID":"8c22cde2-2de5-4760-9d35-8e396f71525c","Type":"ContainerStarted","Data":"d3a435cf46ee2b1fa65d1d657372dbae4d71334222300c60c92667ef459b0c99"} Jul 06 23:56:00.788362 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:00.788353 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:56:00.793308 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:00.793289 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-555fdc4c8c-bwjvn" Jul 06 23:56:00.821457 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:56:00.821422 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" podUID="dc757dda-ca43-4e5a-85bc-926468c9f266" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:56:00.866581 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:00.860849 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc8ff5d88-msr8v"] Jul 06 23:56:00.866581 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:56:00.864150 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-p2qmc\" is forbidden: User \"system:node:ip-10-0-139-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-100.ec2.internal' and this object" podUID="dc757dda-ca43-4e5a-85bc-926468c9f266" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p2qmc" Jul 06 23:56:01.805450 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:01.805418 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc757dda-ca43-4e5a-85bc-926468c9f266" path="/var/lib/kubelet/pods/dc757dda-ca43-4e5a-85bc-926468c9f266/volumes" Jul 06 23:56:04.809659 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:04.809576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" event={"ID":"8c22cde2-2de5-4760-9d35-8e396f71525c","Type":"ContainerStarted","Data":"ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637"} Jul 06 23:56:04.809659 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:04.809633 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:56:04.830767 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:04.830714 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" podStartSLOduration=1.458732465 podStartE2EDuration="5.830699522s" totalImagesPullingTime="4.371967057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:04.829106355 +0000 UTC m=+429.479528302" watchObservedRunningTime="2026-07-06 23:56:04.830699522 +0000 UTC m=+429.481121516" Jul 06 23:56:15.817221 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:15.817194 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 06 23:56:25.896455 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:25.896396 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6dc8ff5d88-msr8v" podUID="04db2e9e-608c-4af4-858c-4d1191e59b87" containerName="console" containerID="cri-o://e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194" gracePeriod=15 Jul 06 23:56:26.146388 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.146365 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:56:26.289373 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289295 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-serving-cert\") pod \"04db2e9e-608c-4af4-858c-4d1191e59b87\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " Jul 06 23:56:26.289373 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289326 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-service-ca\") pod \"04db2e9e-608c-4af4-858c-4d1191e59b87\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " Jul 06 23:56:26.289373 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289364 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-oauth-config\") pod \"04db2e9e-608c-4af4-858c-4d1191e59b87\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " Jul 06 23:56:26.289689 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289479 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-oauth-serving-cert\") pod \"04db2e9e-608c-4af4-858c-4d1191e59b87\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " Jul 06 23:56:26.289689 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289529 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-console-config\") pod \"04db2e9e-608c-4af4-858c-4d1191e59b87\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " Jul 06 23:56:26.289689 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289556 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-trusted-ca-bundle\") pod \"04db2e9e-608c-4af4-858c-4d1191e59b87\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " Jul 06 23:56:26.289689 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289618 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/04db2e9e-608c-4af4-858c-4d1191e59b87-kube-api-access-7df54\") pod \"04db2e9e-608c-4af4-858c-4d1191e59b87\" (UID: \"04db2e9e-608c-4af4-858c-4d1191e59b87\") " Jul 06 23:56:26.289887 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289759 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-service-ca" (OuterVolumeSpecName: "service-ca") pod "04db2e9e-608c-4af4-858c-4d1191e59b87" (UID: "04db2e9e-608c-4af4-858c-4d1191e59b87"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:56:26.289887 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289840 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04db2e9e-608c-4af4-858c-4d1191e59b87" (UID: "04db2e9e-608c-4af4-858c-4d1191e59b87"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:56:26.289988 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289934 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-console-config" (OuterVolumeSpecName: "console-config") pod "04db2e9e-608c-4af4-858c-4d1191e59b87" (UID: "04db2e9e-608c-4af4-858c-4d1191e59b87"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:56:26.290044 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.289980 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "04db2e9e-608c-4af4-858c-4d1191e59b87" (UID: "04db2e9e-608c-4af4-858c-4d1191e59b87"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:56:26.290044 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.290026 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:26.290044 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.290041 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:26.290165 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.290051 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:26.291670 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.291641 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04db2e9e-608c-4af4-858c-4d1191e59b87" (UID: "04db2e9e-608c-4af4-858c-4d1191e59b87"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:56:26.291670 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.291659 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04db2e9e-608c-4af4-858c-4d1191e59b87" (UID: "04db2e9e-608c-4af4-858c-4d1191e59b87"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:56:26.292005 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.291976 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04db2e9e-608c-4af4-858c-4d1191e59b87-kube-api-access-7df54" (OuterVolumeSpecName: "kube-api-access-7df54") pod "04db2e9e-608c-4af4-858c-4d1191e59b87" (UID: "04db2e9e-608c-4af4-858c-4d1191e59b87"). InnerVolumeSpecName "kube-api-access-7df54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:56:26.391043 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.391006 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7df54\" (UniqueName: \"kubernetes.io/projected/04db2e9e-608c-4af4-858c-4d1191e59b87-kube-api-access-7df54\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:26.391043 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.391041 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:26.391043 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.391051 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04db2e9e-608c-4af4-858c-4d1191e59b87-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:26.391234 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.391059 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04db2e9e-608c-4af4-858c-4d1191e59b87-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:26.913851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.913819 2564 generic.go:356] "Generic (PLEG): container finished" podID="04db2e9e-608c-4af4-858c-4d1191e59b87" containerID="e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194" exitCode=0 Jul 06 23:56:26.913851 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.913854 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc8ff5d88-msr8v" Jul 06 23:56:26.914311 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.913869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8ff5d88-msr8v" event={"ID":"04db2e9e-608c-4af4-858c-4d1191e59b87","Type":"ContainerDied","Data":"e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194"} Jul 06 23:56:26.914311 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.913905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc8ff5d88-msr8v" event={"ID":"04db2e9e-608c-4af4-858c-4d1191e59b87","Type":"ContainerDied","Data":"164def0aa676d4e1e9d20c6b6978a48ce391de1fad81dbea44209351b5e6f359"} Jul 06 23:56:26.914311 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.913923 2564 scope.go:117] "RemoveContainer" containerID="e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194" Jul 06 23:56:26.922839 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.922825 2564 scope.go:117] "RemoveContainer" containerID="e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194" Jul 06 23:56:26.923082 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:56:26.923065 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194\": container with ID starting with e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194 not found: ID does not exist" containerID="e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194" Jul 06 23:56:26.923132 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.923089 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194"} err="failed to get container status \"e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194\": rpc error: code = NotFound desc = could not find container \"e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194\": container with ID starting with e713b97d3da86e9f5f28a33968cfa5bd5caecaa64c678a5b0a9af0e6a7c11194 not found: ID does not exist" Jul 06 23:56:26.933656 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.933635 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc8ff5d88-msr8v"] Jul 06 23:56:26.936981 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:26.936963 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dc8ff5d88-msr8v"] Jul 06 23:56:27.805410 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:27.805382 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04db2e9e-608c-4af4-858c-4d1191e59b87" path="/var/lib/kubelet/pods/04db2e9e-608c-4af4-858c-4d1191e59b87/volumes" Jul 06 23:56:36.824290 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.824258 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-mwm42"] Jul 06 23:56:36.824690 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.824658 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04db2e9e-608c-4af4-858c-4d1191e59b87" containerName="console" Jul 06 23:56:36.824690 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.824670 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="04db2e9e-608c-4af4-858c-4d1191e59b87" containerName="console" Jul 06 23:56:36.824759 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.824733 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="04db2e9e-608c-4af4-858c-4d1191e59b87" containerName="console" Jul 06 23:56:36.829662 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.829635 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mwm42" Jul 06 23:56:36.831671 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.831646 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5sqk\"" Jul 06 23:56:36.832416 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.832392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mwm42"] Jul 06 23:56:36.873281 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.873242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvt4\" (UniqueName: \"kubernetes.io/projected/39aa4510-189b-48ac-baf2-26acd58b72cc-kube-api-access-gpvt4\") pod \"authorino-58cd955675-mwm42\" (UID: \"39aa4510-189b-48ac-baf2-26acd58b72cc\") " pod="kuadrant-system/authorino-58cd955675-mwm42" Jul 06 23:56:36.974460 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.974427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvt4\" (UniqueName: \"kubernetes.io/projected/39aa4510-189b-48ac-baf2-26acd58b72cc-kube-api-access-gpvt4\") pod \"authorino-58cd955675-mwm42\" (UID: \"39aa4510-189b-48ac-baf2-26acd58b72cc\") " pod="kuadrant-system/authorino-58cd955675-mwm42" Jul 06 23:56:36.982392 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:36.982368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvt4\" (UniqueName: \"kubernetes.io/projected/39aa4510-189b-48ac-baf2-26acd58b72cc-kube-api-access-gpvt4\") pod \"authorino-58cd955675-mwm42\" (UID: \"39aa4510-189b-48ac-baf2-26acd58b72cc\") " pod="kuadrant-system/authorino-58cd955675-mwm42" Jul 06 23:56:37.153202 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:37.153170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mwm42" Jul 06 23:56:37.288579 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:37.288530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mwm42"] Jul 06 23:56:37.290953 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:56:37.290919 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39aa4510_189b_48ac_baf2_26acd58b72cc.slice/crio-e63286851672070e841cc08c10ba573cb46f7ba735b29b5b2b8722c4c2a395cf WatchSource:0}: Error finding container e63286851672070e841cc08c10ba573cb46f7ba735b29b5b2b8722c4c2a395cf: Status 404 returned error can't find the container with id e63286851672070e841cc08c10ba573cb46f7ba735b29b5b2b8722c4c2a395cf Jul 06 23:56:37.972853 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:37.972813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mwm42" event={"ID":"39aa4510-189b-48ac-baf2-26acd58b72cc","Type":"ContainerStarted","Data":"e63286851672070e841cc08c10ba573cb46f7ba735b29b5b2b8722c4c2a395cf"} Jul 06 23:56:40.992631 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:40.992592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mwm42" event={"ID":"39aa4510-189b-48ac-baf2-26acd58b72cc","Type":"ContainerStarted","Data":"592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3"} Jul 06 23:56:41.008589 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:41.008543 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-mwm42" podStartSLOduration=1.8829560010000002 podStartE2EDuration="5.008527756s" totalImagesPullingTime="3.125571755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:41.00699888 +0000 UTC m=+465.657420828" watchObservedRunningTime="2026-07-06 23:56:41.008527756 +0000 UTC m=+465.658949702" Jul 06 23:56:42.126368 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:42.126330 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mwm42"] Jul 06 23:56:43.000956 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:43.000919 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-mwm42" podUID="39aa4510-189b-48ac-baf2-26acd58b72cc" containerName="authorino" containerID="cri-o://592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3" gracePeriod=30 Jul 06 23:56:43.259082 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:43.259020 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mwm42" Jul 06 23:56:43.337850 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:43.337819 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpvt4\" (UniqueName: \"kubernetes.io/projected/39aa4510-189b-48ac-baf2-26acd58b72cc-kube-api-access-gpvt4\") pod \"39aa4510-189b-48ac-baf2-26acd58b72cc\" (UID: \"39aa4510-189b-48ac-baf2-26acd58b72cc\") " Jul 06 23:56:43.340098 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:43.340068 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aa4510-189b-48ac-baf2-26acd58b72cc-kube-api-access-gpvt4" (OuterVolumeSpecName: "kube-api-access-gpvt4") pod "39aa4510-189b-48ac-baf2-26acd58b72cc" (UID: "39aa4510-189b-48ac-baf2-26acd58b72cc"). InnerVolumeSpecName "kube-api-access-gpvt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:56:43.439232 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:43.439188 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvt4\" (UniqueName: \"kubernetes.io/projected/39aa4510-189b-48ac-baf2-26acd58b72cc-kube-api-access-gpvt4\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:56:44.006207 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.006178 2564 generic.go:356] "Generic (PLEG): container finished" podID="39aa4510-189b-48ac-baf2-26acd58b72cc" containerID="592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3" exitCode=0 Jul 06 23:56:44.006350 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.006219 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mwm42" Jul 06 23:56:44.006350 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.006233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mwm42" event={"ID":"39aa4510-189b-48ac-baf2-26acd58b72cc","Type":"ContainerDied","Data":"592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3"} Jul 06 23:56:44.006350 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.006261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mwm42" event={"ID":"39aa4510-189b-48ac-baf2-26acd58b72cc","Type":"ContainerDied","Data":"e63286851672070e841cc08c10ba573cb46f7ba735b29b5b2b8722c4c2a395cf"} Jul 06 23:56:44.006350 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.006277 2564 scope.go:117] "RemoveContainer" containerID="592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3" Jul 06 23:56:44.014778 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.014761 2564 scope.go:117] "RemoveContainer" containerID="592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3" Jul 06 23:56:44.015040 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:56:44.015022 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3\": container with ID starting with 592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3 not found: ID does not exist" containerID="592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3" Jul 06 23:56:44.015091 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.015052 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3"} err="failed to get container status \"592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3\": rpc error: code = NotFound desc = could not find container \"592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3\": container with ID starting with 592bd4a3f0cdbe87e91d7afb3f3e4eeb572af955ae6874baafb0eb233ca205b3 not found: ID does not exist" Jul 06 23:56:44.022029 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.022007 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mwm42"] Jul 06 23:56:44.025470 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:44.025450 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-mwm42"] Jul 06 23:56:45.806458 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:56:45.806426 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39aa4510-189b-48ac-baf2-26acd58b72cc" path="/var/lib/kubelet/pods/39aa4510-189b-48ac-baf2-26acd58b72cc/volumes" Jul 06 23:57:09.730049 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.730016 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hjsrm"] Jul 06 23:57:09.730477 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.730440 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39aa4510-189b-48ac-baf2-26acd58b72cc" containerName="authorino" Jul 06 23:57:09.730477 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.730455 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39aa4510-189b-48ac-baf2-26acd58b72cc" containerName="authorino" Jul 06 23:57:09.730588 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.730533 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="39aa4510-189b-48ac-baf2-26acd58b72cc" containerName="authorino" Jul 06 23:57:09.734739 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.734720 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hjsrm" Jul 06 23:57:09.736645 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.736619 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5sqk\"" Jul 06 23:57:09.740969 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.740943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hjsrm"] Jul 06 23:57:09.876153 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.876123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x6j7\" (UniqueName: \"kubernetes.io/projected/fc2b0ada-8280-462c-9a3d-016beecd7a3b-kube-api-access-2x6j7\") pod \"authorino-5d4b67c446-hjsrm\" (UID: \"fc2b0ada-8280-462c-9a3d-016beecd7a3b\") " pod="kuadrant-system/authorino-5d4b67c446-hjsrm" Jul 06 23:57:09.914492 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.914467 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hjsrm"] Jul 06 23:57:09.914758 ip-10-0-139-100 kubenswrapper[2564]: E0706 23:57:09.914738 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2x6j7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-hjsrm" podUID="fc2b0ada-8280-462c-9a3d-016beecd7a3b" Jul 06 23:57:09.976956 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.976924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x6j7\" (UniqueName: \"kubernetes.io/projected/fc2b0ada-8280-462c-9a3d-016beecd7a3b-kube-api-access-2x6j7\") pod \"authorino-5d4b67c446-hjsrm\" (UID: \"fc2b0ada-8280-462c-9a3d-016beecd7a3b\") " pod="kuadrant-system/authorino-5d4b67c446-hjsrm" Jul 06 23:57:09.984522 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:09.984448 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x6j7\" (UniqueName: \"kubernetes.io/projected/fc2b0ada-8280-462c-9a3d-016beecd7a3b-kube-api-access-2x6j7\") pod \"authorino-5d4b67c446-hjsrm\" (UID: \"fc2b0ada-8280-462c-9a3d-016beecd7a3b\") " pod="kuadrant-system/authorino-5d4b67c446-hjsrm" Jul 06 23:57:10.043049 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.043019 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f67fbfd98-4j7wv"] Jul 06 23:57:10.046529 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.046489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.048774 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.048746 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 23:57:10.057283 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.057258 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f67fbfd98-4j7wv"] Jul 06 23:57:10.133663 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.133638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hjsrm" Jul 06 23:57:10.161993 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.161955 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hjsrm" Jul 06 23:57:10.178952 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.178926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz52t\" (UniqueName: \"kubernetes.io/projected/23515a45-0fad-4137-8fb6-3e39b59654c7-kube-api-access-mz52t\") pod \"authorino-5f67fbfd98-4j7wv\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.179072 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.178987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23515a45-0fad-4137-8fb6-3e39b59654c7-tls-cert\") pod \"authorino-5f67fbfd98-4j7wv\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.279789 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.279710 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x6j7\" (UniqueName: \"kubernetes.io/projected/fc2b0ada-8280-462c-9a3d-016beecd7a3b-kube-api-access-2x6j7\") pod \"fc2b0ada-8280-462c-9a3d-016beecd7a3b\" (UID: \"fc2b0ada-8280-462c-9a3d-016beecd7a3b\") " Jul 06 23:57:10.279932 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.279839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23515a45-0fad-4137-8fb6-3e39b59654c7-tls-cert\") pod \"authorino-5f67fbfd98-4j7wv\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.279996 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.279947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz52t\" (UniqueName: \"kubernetes.io/projected/23515a45-0fad-4137-8fb6-3e39b59654c7-kube-api-access-mz52t\") pod \"authorino-5f67fbfd98-4j7wv\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.281917 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.281895 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2b0ada-8280-462c-9a3d-016beecd7a3b-kube-api-access-2x6j7" (OuterVolumeSpecName: "kube-api-access-2x6j7") pod "fc2b0ada-8280-462c-9a3d-016beecd7a3b" (UID: "fc2b0ada-8280-462c-9a3d-016beecd7a3b"). InnerVolumeSpecName "kube-api-access-2x6j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:57:10.282355 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.282333 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23515a45-0fad-4137-8fb6-3e39b59654c7-tls-cert\") pod \"authorino-5f67fbfd98-4j7wv\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.296930 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.296904 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz52t\" (UniqueName: \"kubernetes.io/projected/23515a45-0fad-4137-8fb6-3e39b59654c7-kube-api-access-mz52t\") pod \"authorino-5f67fbfd98-4j7wv\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.356027 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.355996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 06 23:57:10.381677 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.381645 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x6j7\" (UniqueName: \"kubernetes.io/projected/fc2b0ada-8280-462c-9a3d-016beecd7a3b-kube-api-access-2x6j7\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 06 23:57:10.496587 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:10.496559 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f67fbfd98-4j7wv"] Jul 06 23:57:10.499087 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:57:10.499062 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23515a45_0fad_4137_8fb6_3e39b59654c7.slice/crio-8a16b08b70d4501a0735d3a58cd4a4601473d01a38c96cde5b1110e482c6a57e WatchSource:0}: Error finding container 8a16b08b70d4501a0735d3a58cd4a4601473d01a38c96cde5b1110e482c6a57e: Status 404 returned error can't find the container with id 8a16b08b70d4501a0735d3a58cd4a4601473d01a38c96cde5b1110e482c6a57e Jul 06 23:57:11.139780 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:11.139747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" event={"ID":"23515a45-0fad-4137-8fb6-3e39b59654c7","Type":"ContainerStarted","Data":"8a16b08b70d4501a0735d3a58cd4a4601473d01a38c96cde5b1110e482c6a57e"} Jul 06 23:57:11.140071 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:11.139803 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hjsrm" Jul 06 23:57:11.168466 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:11.168442 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hjsrm"] Jul 06 23:57:11.173450 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:11.173427 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hjsrm"] Jul 06 23:57:11.806081 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:11.806049 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2b0ada-8280-462c-9a3d-016beecd7a3b" path="/var/lib/kubelet/pods/fc2b0ada-8280-462c-9a3d-016beecd7a3b/volumes" Jul 06 23:57:12.146126 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:12.146093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" event={"ID":"23515a45-0fad-4137-8fb6-3e39b59654c7","Type":"ContainerStarted","Data":"840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63"} Jul 06 23:57:12.161543 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:57:12.161477 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" podStartSLOduration=1.627703721 podStartE2EDuration="2.161463172s" totalImagesPullingTime="533.759451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:12.160791548 +0000 UTC m=+496.811213493" watchObservedRunningTime="2026-07-06 23:57:12.161463172 +0000 UTC m=+496.811885119" Jul 06 23:58:22.092253 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.092216 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq"] Jul 06 23:58:22.095643 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.095626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.098459 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.098440 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 23:58:22.098617 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.098440 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 23:58:22.098617 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.098476 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sb2d5\"" Jul 06 23:58:22.098617 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.098468 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 23:58:22.105491 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.105467 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq"] Jul 06 23:58:22.204915 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.204887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.205244 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.204938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.205244 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.204958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9gl\" (UniqueName: \"kubernetes.io/projected/d5a60c67-6895-49d9-9b8a-26bc3ba91804-kube-api-access-bf9gl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.205244 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.205054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.205244 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.205104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a60c67-6895-49d9-9b8a-26bc3ba91804-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.205244 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.205128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.306630 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.306596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a60c67-6895-49d9-9b8a-26bc3ba91804-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.306630 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.306635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.306864 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.306684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.306864 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.306739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.306864 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.306770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9gl\" (UniqueName: \"kubernetes.io/projected/d5a60c67-6895-49d9-9b8a-26bc3ba91804-kube-api-access-bf9gl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.306864 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.306830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.307080 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.307022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.307138 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.307107 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.307210 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.307191 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.308990 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.308971 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a60c67-6895-49d9-9b8a-26bc3ba91804-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.309249 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.309230 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a60c67-6895-49d9-9b8a-26bc3ba91804-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.314472 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.314450 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9gl\" (UniqueName: \"kubernetes.io/projected/d5a60c67-6895-49d9-9b8a-26bc3ba91804-kube-api-access-bf9gl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b4xtq\" (UID: \"d5a60c67-6895-49d9-9b8a-26bc3ba91804\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.406620 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.406588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:22.545733 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:22.545678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq"] Jul 06 23:58:22.548113 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:58:22.548084 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a60c67_6895_49d9_9b8a_26bc3ba91804.slice/crio-33f56382287746a7c065b1f6aa570a44e7e5c36584c0a827390c76b15efeaa17 WatchSource:0}: Error finding container 33f56382287746a7c065b1f6aa570a44e7e5c36584c0a827390c76b15efeaa17: Status 404 returned error can't find the container with id 33f56382287746a7c065b1f6aa570a44e7e5c36584c0a827390c76b15efeaa17 Jul 06 23:58:23.485921 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:23.485880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" event={"ID":"d5a60c67-6895-49d9-9b8a-26bc3ba91804","Type":"ContainerStarted","Data":"33f56382287746a7c065b1f6aa570a44e7e5c36584c0a827390c76b15efeaa17"} Jul 06 23:58:24.491899 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:24.491858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" event={"ID":"d5a60c67-6895-49d9-9b8a-26bc3ba91804","Type":"ContainerStarted","Data":"35ac650e6c1fca3d956ba77eb72e815eee4781e287ee8193b77ea5170f136168"} Jul 06 23:58:24.509303 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:24.509257 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" podStartSLOduration=1.415184618 podStartE2EDuration="2.509242021s" totalImagesPullingTime="1.094057403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:24.507592497 +0000 UTC m=+569.158014469" watchObservedRunningTime="2026-07-06 23:58:24.509242021 +0000 UTC m=+569.159663968" Jul 06 23:58:32.406866 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:32.406832 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:32.407434 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:32.406898 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:32.419673 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:32.419649 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:32.541481 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:32.541455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b4xtq" Jul 06 23:58:55.719257 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:55.719228 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 06 23:58:55.722082 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:55.722054 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 06 23:58:58.988048 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:58.988015 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m"] Jul 06 23:58:58.991542 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:58.991524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:58.993541 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:58.993520 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 23:58:58.997876 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:58.997858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m"] Jul 06 23:58:59.028022 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.027996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.028114 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.028033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.028114 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.028086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqph\" (UniqueName: \"kubernetes.io/projected/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-kube-api-access-xvqph\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.028114 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.028111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.028223 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.028138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.028223 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.028190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.128894 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.128869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129015 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.128927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129015 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.128967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129123 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.129014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqph\" (UniqueName: \"kubernetes.io/projected/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-kube-api-access-xvqph\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129123 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.129041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129123 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.129082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129445 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.129411 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129445 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.129440 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.129662 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.129463 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.131331 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.131311 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.131421 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.131403 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.136623 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.136602 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqph\" (UniqueName: \"kubernetes.io/projected/23e5d46c-45b8-4c5c-ab59-b6a77ea96fce-kube-api-access-xvqph\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m\" (UID: \"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.302744 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.302658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:58:59.440042 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.440015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m"] Jul 06 23:58:59.442009 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:58:59.441977 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e5d46c_45b8_4c5c_ab59_b6a77ea96fce.slice/crio-a966f21c97bb24241c8fe0416fc77e4c8bc7d5fec7bb03f11608072bb0fc32d2 WatchSource:0}: Error finding container a966f21c97bb24241c8fe0416fc77e4c8bc7d5fec7bb03f11608072bb0fc32d2: Status 404 returned error can't find the container with id a966f21c97bb24241c8fe0416fc77e4c8bc7d5fec7bb03f11608072bb0fc32d2 Jul 06 23:58:59.666696 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:58:59.666670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" event={"ID":"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce","Type":"ContainerStarted","Data":"a966f21c97bb24241c8fe0416fc77e4c8bc7d5fec7bb03f11608072bb0fc32d2"} Jul 06 23:59:00.674479 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:00.674441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" event={"ID":"23e5d46c-45b8-4c5c-ab59-b6a77ea96fce","Type":"ContainerStarted","Data":"cca575f6ba93268eb2e9c8b1a8870d8a3146228fe66f76bfd39dce2dbb22eb80"} Jul 06 23:59:00.689611 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:00.689547 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" podStartSLOduration=2.514830719 podStartE2EDuration="2.689526454s" totalImagesPullingTime="174.695735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:59:00.689406602 +0000 UTC m=+605.339828577" watchObservedRunningTime="2026-07-06 23:59:00.689526454 +0000 UTC m=+605.339948424" Jul 06 23:59:04.171967 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.171936 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h"] Jul 06 23:59:04.175618 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.175598 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.177764 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.177737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 23:59:04.187249 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.187223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h"] Jul 06 23:59:04.279316 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.279286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.279316 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.279317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.279551 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.279343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.279551 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.279443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.279551 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.279510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dql\" (UniqueName: \"kubernetes.io/projected/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-kube-api-access-b8dql\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.279656 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.279573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.380791 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.380760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dql\" (UniqueName: \"kubernetes.io/projected/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-kube-api-access-b8dql\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.380971 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.380823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.380971 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.380897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.380971 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.380920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.380971 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.380956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.381177 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.381001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.381259 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.381226 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.381329 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.381239 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.381329 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.381293 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.383288 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.383259 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.383566 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.383545 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.388188 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.388163 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dql\" (UniqueName: \"kubernetes.io/projected/b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c-kube-api-access-b8dql\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6x46h\" (UID: \"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.488032 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.487961 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:04.623588 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.623557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h"] Jul 06 23:59:04.628797 ip-10-0-139-100 kubenswrapper[2564]: W0706 23:59:04.628767 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e1bc54_e7d7_40f8_8cfc_165e649d5f0c.slice/crio-27ca38fa1b58955cf415520952453f87dcba597a874950e16fb7745dba2f26da WatchSource:0}: Error finding container 27ca38fa1b58955cf415520952453f87dcba597a874950e16fb7745dba2f26da: Status 404 returned error can't find the container with id 27ca38fa1b58955cf415520952453f87dcba597a874950e16fb7745dba2f26da Jul 06 23:59:04.701032 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:04.700997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" event={"ID":"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c","Type":"ContainerStarted","Data":"27ca38fa1b58955cf415520952453f87dcba597a874950e16fb7745dba2f26da"} Jul 06 23:59:05.706988 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:05.706951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" event={"ID":"b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c","Type":"ContainerStarted","Data":"942efbc2a437a40e5149267d927ace28232c4a585328ab4c9902123646122877"} Jul 06 23:59:05.724287 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:05.724243 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" podStartSLOduration=1.54416632 podStartE2EDuration="1.724226385s" totalImagesPullingTime="180.060065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:59:05.722945249 +0000 UTC m=+610.373367231" watchObservedRunningTime="2026-07-06 23:59:05.724226385 +0000 UTC m=+610.374648381" Jul 06 23:59:09.302940 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:09.302903 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:59:09.303416 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:09.302962 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:59:09.315511 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:09.315475 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:59:09.741329 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:09.741297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m" Jul 06 23:59:14.488411 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:14.488379 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:14.488802 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:14.488442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:14.501216 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:14.501190 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 06 23:59:14.766012 ip-10-0-139-100 kubenswrapper[2564]: I0706 23:59:14.765944 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6x46h" Jul 07 00:00:00.240522 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.239833 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f67fbfd98-4j7wv"] Jul 07 00:00:00.240522 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.240062 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" podUID="23515a45-0fad-4137-8fb6-3e39b59654c7" containerName="authorino" containerID="cri-o://840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63" gracePeriod=30 Jul 07 00:00:00.509393 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.509363 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 07 00:00:00.600051 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.600025 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz52t\" (UniqueName: \"kubernetes.io/projected/23515a45-0fad-4137-8fb6-3e39b59654c7-kube-api-access-mz52t\") pod \"23515a45-0fad-4137-8fb6-3e39b59654c7\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " Jul 07 00:00:00.600240 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.600062 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23515a45-0fad-4137-8fb6-3e39b59654c7-tls-cert\") pod \"23515a45-0fad-4137-8fb6-3e39b59654c7\" (UID: \"23515a45-0fad-4137-8fb6-3e39b59654c7\") " Jul 07 00:00:00.602317 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.602289 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23515a45-0fad-4137-8fb6-3e39b59654c7-kube-api-access-mz52t" (OuterVolumeSpecName: "kube-api-access-mz52t") pod "23515a45-0fad-4137-8fb6-3e39b59654c7" (UID: "23515a45-0fad-4137-8fb6-3e39b59654c7"). InnerVolumeSpecName "kube-api-access-mz52t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:00:00.610445 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.610415 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23515a45-0fad-4137-8fb6-3e39b59654c7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "23515a45-0fad-4137-8fb6-3e39b59654c7" (UID: "23515a45-0fad-4137-8fb6-3e39b59654c7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:00:00.701112 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.701077 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz52t\" (UniqueName: \"kubernetes.io/projected/23515a45-0fad-4137-8fb6-3e39b59654c7-kube-api-access-mz52t\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:00:00.701112 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.701104 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23515a45-0fad-4137-8fb6-3e39b59654c7-tls-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:00:00.987316 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.987283 2564 generic.go:356] "Generic (PLEG): container finished" podID="23515a45-0fad-4137-8fb6-3e39b59654c7" containerID="840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63" exitCode=0 Jul 07 00:00:00.987529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.987337 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" Jul 07 00:00:00.987529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.987373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" event={"ID":"23515a45-0fad-4137-8fb6-3e39b59654c7","Type":"ContainerDied","Data":"840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63"} Jul 07 00:00:00.987529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.987423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f67fbfd98-4j7wv" event={"ID":"23515a45-0fad-4137-8fb6-3e39b59654c7","Type":"ContainerDied","Data":"8a16b08b70d4501a0735d3a58cd4a4601473d01a38c96cde5b1110e482c6a57e"} Jul 07 00:00:00.987529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.987445 2564 scope.go:117] "RemoveContainer" containerID="840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63" Jul 07 00:00:00.996476 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.996461 2564 scope.go:117] "RemoveContainer" containerID="840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63" Jul 07 00:00:00.996756 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:00:00.996739 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63\": container with ID starting with 840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63 not found: ID does not exist" containerID="840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63" Jul 07 00:00:00.996805 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:00.996765 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63"} err="failed to get container status \"840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63\": rpc error: code = NotFound desc = could not find container \"840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63\": container with ID starting with 840754f0f33d6b9606aeed574c3bb854bb9227095d4d2a49d66f36ed375c9a63 not found: ID does not exist" Jul 07 00:00:01.008820 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:01.008795 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f67fbfd98-4j7wv"] Jul 07 00:00:01.012157 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:01.012136 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f67fbfd98-4j7wv"] Jul 07 00:00:01.806189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:00:01.806156 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23515a45-0fad-4137-8fb6-3e39b59654c7" path="/var/lib/kubelet/pods/23515a45-0fad-4137-8fb6-3e39b59654c7/volumes" Jul 07 00:03:55.752361 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:03:55.752293 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:03:55.753445 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:03:55.753421 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:05:55.193664 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:55.193603 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b"] Jul 07 00:05:55.194269 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:55.193854 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" podUID="8c22cde2-2de5-4760-9d35-8e396f71525c" containerName="manager" containerID="cri-o://ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637" gracePeriod=10 Jul 07 00:05:57.455103 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.455077 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 07 00:05:57.574837 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.574742 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c22cde2-2de5-4760-9d35-8e396f71525c-extensions-socket-volume\") pod \"8c22cde2-2de5-4760-9d35-8e396f71525c\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " Jul 07 00:05:57.574837 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.574790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krcd4\" (UniqueName: \"kubernetes.io/projected/8c22cde2-2de5-4760-9d35-8e396f71525c-kube-api-access-krcd4\") pod \"8c22cde2-2de5-4760-9d35-8e396f71525c\" (UID: \"8c22cde2-2de5-4760-9d35-8e396f71525c\") " Jul 07 00:05:57.575147 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.575122 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c22cde2-2de5-4760-9d35-8e396f71525c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8c22cde2-2de5-4760-9d35-8e396f71525c" (UID: "8c22cde2-2de5-4760-9d35-8e396f71525c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:57.577029 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.577005 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c22cde2-2de5-4760-9d35-8e396f71525c-kube-api-access-krcd4" (OuterVolumeSpecName: "kube-api-access-krcd4") pod "8c22cde2-2de5-4760-9d35-8e396f71525c" (UID: "8c22cde2-2de5-4760-9d35-8e396f71525c"). InnerVolumeSpecName "kube-api-access-krcd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:05:57.676384 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.676347 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c22cde2-2de5-4760-9d35-8e396f71525c-extensions-socket-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:05:57.676384 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.676371 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krcd4\" (UniqueName: \"kubernetes.io/projected/8c22cde2-2de5-4760-9d35-8e396f71525c-kube-api-access-krcd4\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:05:57.795701 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.795659 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c22cde2-2de5-4760-9d35-8e396f71525c" containerID="ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637" exitCode=0 Jul 07 00:05:57.795907 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.795707 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" Jul 07 00:05:57.795907 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.795747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" event={"ID":"8c22cde2-2de5-4760-9d35-8e396f71525c","Type":"ContainerDied","Data":"ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637"} Jul 07 00:05:57.795907 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.795817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b" event={"ID":"8c22cde2-2de5-4760-9d35-8e396f71525c","Type":"ContainerDied","Data":"d3a435cf46ee2b1fa65d1d657372dbae4d71334222300c60c92667ef459b0c99"} Jul 07 00:05:57.795907 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.795851 2564 scope.go:117] "RemoveContainer" containerID="ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637" Jul 07 00:05:57.806620 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.806597 2564 scope.go:117] "RemoveContainer" containerID="ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637" Jul 07 00:05:57.806874 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:05:57.806855 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637\": container with ID starting with ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637 not found: ID does not exist" containerID="ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637" Jul 07 00:05:57.806959 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.806882 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637"} err="failed to get container status \"ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637\": rpc error: code = NotFound desc = could not find container \"ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637\": container with ID starting with ad65c82ea45d72d9d04f63af4541560148f8b1a7fcf84fe6ba80e029f8013637 not found: ID does not exist" Jul 07 00:05:57.830584 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.830507 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b"] Jul 07 00:05:57.844708 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:57.844680 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ppg2b"] Jul 07 00:05:59.806574 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:05:59.806540 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c22cde2-2de5-4760-9d35-8e396f71525c" path="/var/lib/kubelet/pods/8c22cde2-2de5-4760-9d35-8e396f71525c/volumes" Jul 07 00:06:56.314363 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.314328 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr"] Jul 07 00:06:56.314762 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.314724 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c22cde2-2de5-4760-9d35-8e396f71525c" containerName="manager" Jul 07 00:06:56.314762 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.314735 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c22cde2-2de5-4760-9d35-8e396f71525c" containerName="manager" Jul 07 00:06:56.314762 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.314753 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23515a45-0fad-4137-8fb6-3e39b59654c7" containerName="authorino" Jul 07 00:06:56.314762 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.314759 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="23515a45-0fad-4137-8fb6-3e39b59654c7" containerName="authorino" Jul 07 00:06:56.314902 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.314823 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="23515a45-0fad-4137-8fb6-3e39b59654c7" containerName="authorino" Jul 07 00:06:56.314902 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.314833 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c22cde2-2de5-4760-9d35-8e396f71525c" containerName="manager" Jul 07 00:06:56.318305 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.318287 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.320789 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.320770 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gk8sk\"" Jul 07 00:06:56.328318 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.328294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr"] Jul 07 00:06:56.410312 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.410270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrs7\" (UniqueName: \"kubernetes.io/projected/78cd98e8-58ab-43c3-9969-0aa78b3df6ed-kube-api-access-wlrs7\") pod \"kuadrant-operator-controller-manager-86878c769f-bpsbr\" (UID: \"78cd98e8-58ab-43c3-9969-0aa78b3df6ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.410485 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.410349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78cd98e8-58ab-43c3-9969-0aa78b3df6ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bpsbr\" (UID: \"78cd98e8-58ab-43c3-9969-0aa78b3df6ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.511678 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.511647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78cd98e8-58ab-43c3-9969-0aa78b3df6ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bpsbr\" (UID: \"78cd98e8-58ab-43c3-9969-0aa78b3df6ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.511843 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.511741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrs7\" (UniqueName: \"kubernetes.io/projected/78cd98e8-58ab-43c3-9969-0aa78b3df6ed-kube-api-access-wlrs7\") pod \"kuadrant-operator-controller-manager-86878c769f-bpsbr\" (UID: \"78cd98e8-58ab-43c3-9969-0aa78b3df6ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.512096 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.512075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78cd98e8-58ab-43c3-9969-0aa78b3df6ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bpsbr\" (UID: \"78cd98e8-58ab-43c3-9969-0aa78b3df6ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.521122 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.521094 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrs7\" (UniqueName: \"kubernetes.io/projected/78cd98e8-58ab-43c3-9969-0aa78b3df6ed-kube-api-access-wlrs7\") pod \"kuadrant-operator-controller-manager-86878c769f-bpsbr\" (UID: \"78cd98e8-58ab-43c3-9969-0aa78b3df6ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.628709 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.628607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:56.777909 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:56.777866 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr"] Jul 07 00:06:56.781014 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:06:56.780988 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cd98e8_58ab_43c3_9969_0aa78b3df6ed.slice/crio-10531938dc43608b8cf4a65c2178ec4201a5fd0b9e006a490dc8e612d208ba66 WatchSource:0}: Error finding container 10531938dc43608b8cf4a65c2178ec4201a5fd0b9e006a490dc8e612d208ba66: Status 404 returned error can't find the container with id 10531938dc43608b8cf4a65c2178ec4201a5fd0b9e006a490dc8e612d208ba66 Jul 07 00:06:57.081948 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:57.081917 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" event={"ID":"78cd98e8-58ab-43c3-9969-0aa78b3df6ed","Type":"ContainerStarted","Data":"b7b9d6624ebd67ddb7c34f4b61a24fdd9c140ecc54c2bfeb6671a6789e24a5b8"} Jul 07 00:06:57.082134 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:57.081981 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:06:57.082134 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:57.082007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" event={"ID":"78cd98e8-58ab-43c3-9969-0aa78b3df6ed","Type":"ContainerStarted","Data":"10531938dc43608b8cf4a65c2178ec4201a5fd0b9e006a490dc8e612d208ba66"} Jul 07 00:06:57.100402 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:06:57.100352 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" podStartSLOduration=1.100334121 podStartE2EDuration="1.100334121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:06:57.099079531 +0000 UTC m=+1081.749501478" watchObservedRunningTime="2026-07-07 00:06:57.100334121 +0000 UTC m=+1081.750756068" Jul 07 00:07:08.089326 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:07:08.089297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bpsbr" Jul 07 00:08:55.789537 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:08:55.789438 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:08:55.791653 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:08:55.791632 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:13:30.603945 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.603910 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr"] Jul 07 00:13:30.607463 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.607440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.609620 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.609600 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-57dc1bd9-openshift-default-dockercfg-rrzns\"" Jul 07 00:13:30.619611 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.619588 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr"] Jul 07 00:13:30.732585 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.732550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-podinfo\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.732841 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.732813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istiod-ca-cert\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.732952 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.732886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-socket\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.732952 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.732941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-envoy\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.733031 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.732966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fzj\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-kube-api-access-m2fzj\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.733113 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.733088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-token\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.733163 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.733148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-credential-socket\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.733239 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.733215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-data\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.733288 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.733265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-certs\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834267 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-token\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-credential-socket\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-data\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-certs\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-podinfo\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834728 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istiod-ca-cert\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834728 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-socket\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834728 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-envoy\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.834728 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.834702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fzj\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-kube-api-access-m2fzj\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.835114 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.835087 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-certs\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.835689 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.835617 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-data\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.835929 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.835904 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-credential-socket\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.836550 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.836343 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istiod-ca-cert\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.836701 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.836572 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-socket\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.837846 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.837818 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-envoy\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.838189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.838172 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-podinfo\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.844071 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.844041 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-token\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.844171 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.844123 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fzj\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-kube-api-access-m2fzj\") pod \"e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:30.921221 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:30.921197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:31.080793 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:31.080733 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr"] Jul 07 00:13:31.083205 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:13:31.083178 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dd3b57_3e3c_4c8b_8c5d_d6b0d8212492.slice/crio-896ab61d6c1807d34422d6e2ce016f0fbec75874be7358ad89ef506472e23d1c WatchSource:0}: Error finding container 896ab61d6c1807d34422d6e2ce016f0fbec75874be7358ad89ef506472e23d1c: Status 404 returned error can't find the container with id 896ab61d6c1807d34422d6e2ce016f0fbec75874be7358ad89ef506472e23d1c Jul 07 00:13:31.085267 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:31.085238 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:13:31.085358 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:31.085298 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:13:31.085358 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:31.085325 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:13:32.072716 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:32.072676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" event={"ID":"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492","Type":"ContainerStarted","Data":"a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187"} Jul 07 00:13:32.073197 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:32.072746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" event={"ID":"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492","Type":"ContainerStarted","Data":"896ab61d6c1807d34422d6e2ce016f0fbec75874be7358ad89ef506472e23d1c"} Jul 07 00:13:32.095642 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:32.095598 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" podStartSLOduration=2.095583327 podStartE2EDuration="2.095583327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:13:32.093919662 +0000 UTC m=+1476.744341618" watchObservedRunningTime="2026-07-07 00:13:32.095583327 +0000 UTC m=+1476.746005275" Jul 07 00:13:32.921817 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:32.921790 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:32.926102 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:32.926081 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:33.077613 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:33.077583 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:33.078601 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:33.078581 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:36.449427 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:36.449381 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr"] Jul 07 00:13:36.449962 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:36.449690 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" podUID="78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" containerName="istio-proxy" containerID="cri-o://a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187" gracePeriod=30 Jul 07 00:13:42.307874 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.307849 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:42.451728 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.451702 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-data\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.451868 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.451745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-token\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.451868 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.451791 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istiod-ca-cert\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.451868 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.451837 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-socket\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.451868 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.451861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-podinfo\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.452050 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452009 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2fzj\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-kube-api-access-m2fzj\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.452106 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452082 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-certs\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.452160 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452096 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:42.452160 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452154 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-envoy\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.452261 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452178 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-credential-socket\") pod \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\" (UID: \"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492\") " Jul 07 00:13:42.452261 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452245 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:13:42.452402 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452382 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:42.452609 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452556 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.452609 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452579 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.452609 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452579 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-data" (OuterVolumeSpecName: "istio-data") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:42.452609 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452593 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.452852 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.452802 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:42.454358 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.454327 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-token" (OuterVolumeSpecName: "istio-token") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:13:42.454459 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.454362 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:13:42.454556 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.454536 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:13:42.454793 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.454771 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-kube-api-access-m2fzj" (OuterVolumeSpecName: "kube-api-access-m2fzj") pod "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" (UID: "78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492"). InnerVolumeSpecName "kube-api-access-m2fzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:13:42.553707 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.553676 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2fzj\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-kube-api-access-m2fzj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.553707 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.553700 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.553707 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.553710 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.553916 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.553719 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.553916 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.553727 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:42.553916 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:42.553736 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:13:43.133290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.133257 2564 generic.go:356] "Generic (PLEG): container finished" podID="78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" containerID="a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187" exitCode=0 Jul 07 00:13:43.133471 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.133308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" event={"ID":"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492","Type":"ContainerDied","Data":"a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187"} Jul 07 00:13:43.133471 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.133318 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" Jul 07 00:13:43.133471 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.133350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr" event={"ID":"78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492","Type":"ContainerDied","Data":"896ab61d6c1807d34422d6e2ce016f0fbec75874be7358ad89ef506472e23d1c"} Jul 07 00:13:43.133471 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.133373 2564 scope.go:117] "RemoveContainer" containerID="a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187" Jul 07 00:13:43.143697 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.143677 2564 scope.go:117] "RemoveContainer" containerID="a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187" Jul 07 00:13:43.144344 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:13:43.143970 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187\": container with ID starting with a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187 not found: ID does not exist" containerID="a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187" Jul 07 00:13:43.144344 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.144012 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187"} err="failed to get container status \"a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187\": rpc error: code = NotFound desc = could not find container \"a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187\": container with ID starting with a5b4d9686cb7c7761a30ef50a896805e53629f9ff938b43579eb6c76fc278187 not found: ID does not exist" Jul 07 00:13:43.159019 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.158993 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr"] Jul 07 00:13:43.162680 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.162647 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-57dc1bd9-openshift-default-7dd687f76c-4gflr"] Jul 07 00:13:43.806037 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:43.806007 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" path="/var/lib/kubelet/pods/78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492/volumes" Jul 07 00:13:55.831330 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:55.831305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:13:55.833061 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:13:55.833038 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:16:16.376739 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.376703 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh"] Jul 07 00:16:16.377138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.377094 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" containerName="istio-proxy" Jul 07 00:16:16.377138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.377105 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" containerName="istio-proxy" Jul 07 00:16:16.377209 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.377194 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78dd3b57-3e3c-4c8b-8c5d-d6b0d8212492" containerName="istio-proxy" Jul 07 00:16:16.380492 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.380468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.382764 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.382743 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-104fd30e\"/\"openshift-service-ca.crt\"" Jul 07 00:16:16.383562 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.383544 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-104fd30e\"/\"e2e-lifecycle-model-104fd30e-kserve-self-signed-certs\"" Jul 07 00:16:16.383562 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.383558 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-104fd30e\"/\"kube-root-ca.crt\"" Jul 07 00:16:16.383697 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.383557 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-104fd30e\"/\"default-dockercfg-pxxfs\"" Jul 07 00:16:16.388297 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.388280 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh"] Jul 07 00:16:16.508025 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.507995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-dshm\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.508025 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.508028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tmp-dir\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.508241 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.508055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tls-certs\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.508241 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.508161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-model-cache\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.508241 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.508206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-home\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.508343 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.508273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkkk8\" (UniqueName: \"kubernetes.io/projected/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-kube-api-access-vkkk8\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609405 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-dshm\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609405 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tmp-dir\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609669 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tls-certs\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609669 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-model-cache\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609669 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-home\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609669 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkkk8\" (UniqueName: \"kubernetes.io/projected/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-kube-api-access-vkkk8\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609923 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609897 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tmp-dir\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.609923 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609915 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-model-cache\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.610018 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.609997 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-home\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.611819 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.611788 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-dshm\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.612177 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.612152 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tls-certs\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.617248 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.617230 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkkk8\" (UniqueName: \"kubernetes.io/projected/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-kube-api-access-vkkk8\") pod \"e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.700669 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.700636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:16.840916 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.840888 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh"] Jul 07 00:16:16.842950 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:16:16.842925 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod971d3a72_04d5_4ff4_bdf0_72bfad2ff408.slice/crio-85d504e5d5e04a2d4c348990bdb907a064b910ec8ae6f89da1ac7071621fc739 WatchSource:0}: Error finding container 85d504e5d5e04a2d4c348990bdb907a064b910ec8ae6f89da1ac7071621fc739: Status 404 returned error can't find the container with id 85d504e5d5e04a2d4c348990bdb907a064b910ec8ae6f89da1ac7071621fc739 Jul 07 00:16:16.890749 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:16.890723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" event={"ID":"971d3a72-04d5-4ff4-bdf0-72bfad2ff408","Type":"ContainerStarted","Data":"85d504e5d5e04a2d4c348990bdb907a064b910ec8ae6f89da1ac7071621fc739"} Jul 07 00:16:17.896770 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:17.896735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" event={"ID":"971d3a72-04d5-4ff4-bdf0-72bfad2ff408","Type":"ContainerStarted","Data":"0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06"} Jul 07 00:16:17.914636 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:17.914583 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" podStartSLOduration=1.9145679979999999 podStartE2EDuration="1.914567998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:16:17.912751262 +0000 UTC m=+1642.563173221" watchObservedRunningTime="2026-07-07 00:16:17.914567998 +0000 UTC m=+1642.564989944" Jul 07 00:16:26.700849 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:26.700812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:26.701243 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:26.700861 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:26.713206 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:26.713180 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:26.953314 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:26.953221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:16:51.324652 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:16:51.324616 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh"] Jul 07 00:17:06.345755 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.345702 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" podUID="971d3a72-04d5-4ff4-bdf0-72bfad2ff408" containerName="main" containerID="cri-o://0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06" gracePeriod=45 Jul 07 00:17:06.592892 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.592864 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:17:06.761622 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761590 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-home\") pod \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " Jul 07 00:17:06.761787 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761629 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tmp-dir\") pod \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " Jul 07 00:17:06.761787 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761669 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkkk8\" (UniqueName: \"kubernetes.io/projected/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-kube-api-access-vkkk8\") pod \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " Jul 07 00:17:06.761787 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761695 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tls-certs\") pod \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " Jul 07 00:17:06.761787 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761729 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-dshm\") pod \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " Jul 07 00:17:06.761787 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-model-cache\") pod \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\" (UID: \"971d3a72-04d5-4ff4-bdf0-72bfad2ff408\") " Jul 07 00:17:06.762059 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761901 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-home" (OuterVolumeSpecName: "home") pod "971d3a72-04d5-4ff4-bdf0-72bfad2ff408" (UID: "971d3a72-04d5-4ff4-bdf0-72bfad2ff408"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:17:06.762059 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.761916 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "971d3a72-04d5-4ff4-bdf0-72bfad2ff408" (UID: "971d3a72-04d5-4ff4-bdf0-72bfad2ff408"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:17:06.762059 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.762048 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:17:06.762195 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.762064 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:17:06.762195 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.762077 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-model-cache" (OuterVolumeSpecName: "model-cache") pod "971d3a72-04d5-4ff4-bdf0-72bfad2ff408" (UID: "971d3a72-04d5-4ff4-bdf0-72bfad2ff408"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:17:06.764036 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.764009 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-dshm" (OuterVolumeSpecName: "dshm") pod "971d3a72-04d5-4ff4-bdf0-72bfad2ff408" (UID: "971d3a72-04d5-4ff4-bdf0-72bfad2ff408"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:17:06.764114 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.764073 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "971d3a72-04d5-4ff4-bdf0-72bfad2ff408" (UID: "971d3a72-04d5-4ff4-bdf0-72bfad2ff408"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:17:06.764150 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.764119 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-kube-api-access-vkkk8" (OuterVolumeSpecName: "kube-api-access-vkkk8") pod "971d3a72-04d5-4ff4-bdf0-72bfad2ff408" (UID: "971d3a72-04d5-4ff4-bdf0-72bfad2ff408"). InnerVolumeSpecName "kube-api-access-vkkk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:17:06.862903 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.862878 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkkk8\" (UniqueName: \"kubernetes.io/projected/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-kube-api-access-vkkk8\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:17:06.862903 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.862903 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:17:06.863105 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.862915 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:17:06.863105 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:06.862927 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971d3a72-04d5-4ff4-bdf0-72bfad2ff408-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:17:07.139067 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.139039 2564 generic.go:356] "Generic (PLEG): container finished" podID="971d3a72-04d5-4ff4-bdf0-72bfad2ff408" containerID="0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06" exitCode=0 Jul 07 00:17:07.139253 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.139093 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" Jul 07 00:17:07.139253 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.139139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" event={"ID":"971d3a72-04d5-4ff4-bdf0-72bfad2ff408","Type":"ContainerDied","Data":"0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06"} Jul 07 00:17:07.139253 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.139179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh" event={"ID":"971d3a72-04d5-4ff4-bdf0-72bfad2ff408","Type":"ContainerDied","Data":"85d504e5d5e04a2d4c348990bdb907a064b910ec8ae6f89da1ac7071621fc739"} Jul 07 00:17:07.139253 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.139202 2564 scope.go:117] "RemoveContainer" containerID="0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06" Jul 07 00:17:07.148999 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.148983 2564 scope.go:117] "RemoveContainer" containerID="0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06" Jul 07 00:17:07.149243 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:17:07.149225 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06\": container with ID starting with 0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06 not found: ID does not exist" containerID="0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06" Jul 07 00:17:07.149279 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.149252 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06"} err="failed to get container status \"0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06\": rpc error: code = NotFound desc = could not find container \"0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06\": container with ID starting with 0001521f3a18ce2c8b41d84a1dc32c9ea40063be89fda60ee9f43f2f30042b06 not found: ID does not exist" Jul 07 00:17:07.160188 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.160166 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh"] Jul 07 00:17:07.162109 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.162089 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-104fd30e/e2e-lifecycle-model-104fd30e-kserve-59cb687799-gbgwh"] Jul 07 00:17:07.805219 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:17:07.805188 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971d3a72-04d5-4ff4-bdf0-72bfad2ff408" path="/var/lib/kubelet/pods/971d3a72-04d5-4ff4-bdf0-72bfad2ff408/volumes" Jul 07 00:18:51.176860 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.176780 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk"] Jul 07 00:18:51.179415 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.177189 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971d3a72-04d5-4ff4-bdf0-72bfad2ff408" containerName="main" Jul 07 00:18:51.179415 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.177201 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="971d3a72-04d5-4ff4-bdf0-72bfad2ff408" containerName="main" Jul 07 00:18:51.179415 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.177265 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="971d3a72-04d5-4ff4-bdf0-72bfad2ff408" containerName="main" Jul 07 00:18:51.180192 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.180172 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.182296 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.182277 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ea7465-openshift-default-dockercfg-w8mlv\"" Jul 07 00:18:51.190445 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.190419 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk"] Jul 07 00:18:51.236209 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-data\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236339 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-podinfo\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236339 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-envoy\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236339 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-token\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236475 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-certs\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236475 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-credential-socket\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236475 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6f3685-c354-48b6-a996-3c3e26ac001c-istiod-ca-cert\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236628 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-socket\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.236628 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.236575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn27h\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-kube-api-access-pn27h\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.337819 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.337787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-envoy\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.337994 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.337831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-token\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.337994 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.337870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-certs\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338115 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-credential-socket\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338115 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6f3685-c354-48b6-a996-3c3e26ac001c-istiod-ca-cert\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338236 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-socket\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338236 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn27h\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-kube-api-access-pn27h\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338347 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338313 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-certs\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338347 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-data\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338430 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-credential-socket\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338595 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-socket\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338718 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-podinfo\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.338795 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.338760 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-data\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.339109 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.339083 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6f3685-c354-48b6-a996-3c3e26ac001c-istiod-ca-cert\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.340424 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.340397 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-envoy\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.341100 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.341082 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-podinfo\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.345770 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.345749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-token\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.346075 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.346051 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn27h\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-kube-api-access-pn27h\") pod \"e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.491454 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.491383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:51.633991 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.633954 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk"] Jul 07 00:18:51.636900 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:18:51.636867 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6f3685_c354_48b6_a996_3c3e26ac001c.slice/crio-4d45e3fff1a2001c5a1eee6af9a811685bd6a652df816b48e5027d7d8c2782c2 WatchSource:0}: Error finding container 4d45e3fff1a2001c5a1eee6af9a811685bd6a652df816b48e5027d7d8c2782c2: Status 404 returned error can't find the container with id 4d45e3fff1a2001c5a1eee6af9a811685bd6a652df816b48e5027d7d8c2782c2 Jul 07 00:18:51.639385 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.639354 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:18:51.639470 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.639420 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:18:51.639470 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.639451 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:18:51.667080 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:51.667057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" event={"ID":"5a6f3685-c354-48b6-a996-3c3e26ac001c","Type":"ContainerStarted","Data":"4d45e3fff1a2001c5a1eee6af9a811685bd6a652df816b48e5027d7d8c2782c2"} Jul 07 00:18:52.673653 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:52.673620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" event={"ID":"5a6f3685-c354-48b6-a996-3c3e26ac001c","Type":"ContainerStarted","Data":"5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d"} Jul 07 00:18:52.696713 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:52.696650 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" podStartSLOduration=1.696634217 podStartE2EDuration="1.696634217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:18:52.694818661 +0000 UTC m=+1797.345240635" watchObservedRunningTime="2026-07-07 00:18:52.696634217 +0000 UTC m=+1797.347056163" Jul 07 00:18:53.491742 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:53.491711 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:53.497628 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:53.497604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:53.679732 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:53.679698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:53.680610 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:53.680592 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:18:55.861916 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:55.861893 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:18:55.864676 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:18:55.864654 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:19:07.752881 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:07.752804 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerID="5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d" exitCode=0 Jul 07 00:19:07.752881 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:07.752846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" event={"ID":"5a6f3685-c354-48b6-a996-3c3e26ac001c","Type":"ContainerDied","Data":"5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d"} Jul 07 00:19:07.753328 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:07.753230 2564 scope.go:117] "RemoveContainer" containerID="5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d" Jul 07 00:19:07.754234 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:07.754206 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:19:07.754301 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:07.754275 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:19:07.754339 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:07.754311 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:19:08.679769 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:08.679729 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:19:08.763213 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:08.763169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" event={"ID":"5a6f3685-c354-48b6-a996-3c3e26ac001c","Type":"ContainerStarted","Data":"3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878"} Jul 07 00:19:09.492009 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:09.491964 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:19:09.493209 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:09.493179 2564 patch_prober.go:28] interesting pod/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 07 00:19:09.493339 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:09.493242 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 07 00:19:10.491920 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:10.491884 2564 patch_prober.go:28] interesting pod/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 07 00:19:10.492311 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:10.491948 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 07 00:19:11.492098 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:11.492061 2564 patch_prober.go:28] interesting pod/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 07 00:19:11.492481 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:11.492129 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 07 00:19:12.491987 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:12.491953 2564 patch_prober.go:28] interesting pod/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 07 00:19:12.492164 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:12.492004 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 07 00:19:13.496606 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:13.496576 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:19:13.497088 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:13.496845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:19:13.497433 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:13.497415 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:19:17.892955 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.892922 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb"] Jul 07 00:19:17.897119 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.897101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:17.899757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.899734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-474e44\"/\"default-dockercfg-f9n4x\"" Jul 07 00:19:17.899757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.899744 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-474e44\"/\"kube-root-ca.crt\"" Jul 07 00:19:17.899924 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.899763 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-474e44\"/\"test-model-474e44-kserve-self-signed-certs\"" Jul 07 00:19:17.899924 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.899787 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-474e44\"/\"openshift-service-ca.crt\"" Jul 07 00:19:17.906716 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.906695 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb"] Jul 07 00:19:17.918698 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.918668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-dshm\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:17.918843 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.918709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-model-cache\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:17.918843 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.918813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-home\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:17.918976 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.918869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmxf\" (UniqueName: \"kubernetes.io/projected/9186ed0c-1063-4f3e-a37d-5546814b7ea3-kube-api-access-7mmxf\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:17.918976 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.918947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tls-certs\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:17.919127 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:17.919014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tmp-dir\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020243 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-dshm\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020420 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-model-cache\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020420 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-home\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020420 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmxf\" (UniqueName: \"kubernetes.io/projected/9186ed0c-1063-4f3e-a37d-5546814b7ea3-kube-api-access-7mmxf\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020420 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tls-certs\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020420 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tmp-dir\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020808 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020770 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-home\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020926 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020808 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-model-cache\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.020926 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.020825 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tmp-dir\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.022678 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.022650 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-dshm\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.022927 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.022911 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tls-certs\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.028388 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.028368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmxf\" (UniqueName: \"kubernetes.io/projected/9186ed0c-1063-4f3e-a37d-5546814b7ea3-kube-api-access-7mmxf\") pod \"test-model-474e44-kserve-68cc77857c-rpvgb\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.217600 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.217519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:18.359995 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.359970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb"] Jul 07 00:19:18.361780 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:19:18.361757 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9186ed0c_1063_4f3e_a37d_5546814b7ea3.slice/crio-c580487a1654acf4095648d5a596cb3cf26f27f66ae1b418eaef35a4ef228376 WatchSource:0}: Error finding container c580487a1654acf4095648d5a596cb3cf26f27f66ae1b418eaef35a4ef228376: Status 404 returned error can't find the container with id c580487a1654acf4095648d5a596cb3cf26f27f66ae1b418eaef35a4ef228376 Jul 07 00:19:18.813254 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.813225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" event={"ID":"9186ed0c-1063-4f3e-a37d-5546814b7ea3","Type":"ContainerStarted","Data":"129d1f3ae299bee8edee255372a0c550921228585ebe7ca7fcb6a2a3e1c0d151"} Jul 07 00:19:18.813421 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.813262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" event={"ID":"9186ed0c-1063-4f3e-a37d-5546814b7ea3","Type":"ContainerStarted","Data":"c580487a1654acf4095648d5a596cb3cf26f27f66ae1b418eaef35a4ef228376"} Jul 07 00:19:18.829840 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:18.829796 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" podStartSLOduration=1.829780822 podStartE2EDuration="1.829780822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:19:18.828350358 +0000 UTC m=+1823.478772336" watchObservedRunningTime="2026-07-07 00:19:18.829780822 +0000 UTC m=+1823.480202769" Jul 07 00:19:28.217698 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:28.217662 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:28.218161 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:28.217954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:28.231871 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:28.231845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:28.879967 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:28.879936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:19:43.398456 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.398415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km"] Jul 07 00:19:43.402601 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.402577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.404810 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.404786 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea7465\"/\"openshift-service-ca.crt\"" Jul 07 00:19:43.405650 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.405624 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea7465\"/\"kube-root-ca.crt\"" Jul 07 00:19:43.405650 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.405638 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea7465\"/\"default-dockercfg-z8zq9\"" Jul 07 00:19:43.405833 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.405668 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea7465\"/\"test-model-ea7465-kserve-self-signed-certs\"" Jul 07 00:19:43.411491 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.411466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km"] Jul 07 00:19:43.562258 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.562213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-home\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.562258 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.562258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-dshm\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.562539 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.562281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tmp-dir\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.562539 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.562375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tls-certs\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.562539 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.562406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srrzs\" (UniqueName: \"kubernetes.io/projected/1ff9033d-3bb9-4ef4-b5d5-740161efd279-kube-api-access-srrzs\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.562539 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.562450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-model-cache\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.663358 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srrzs\" (UniqueName: \"kubernetes.io/projected/1ff9033d-3bb9-4ef4-b5d5-740161efd279-kube-api-access-srrzs\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.663358 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-model-cache\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.663759 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-home\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.663759 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-dshm\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.663759 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tmp-dir\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.663759 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tls-certs\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.663759 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663667 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-model-cache\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.664036 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663787 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-home\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.664036 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.663894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tmp-dir\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.665993 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.665965 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-dshm\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.666145 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.666130 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tls-certs\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.672947 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.672926 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srrzs\" (UniqueName: \"kubernetes.io/projected/1ff9033d-3bb9-4ef4-b5d5-740161efd279-kube-api-access-srrzs\") pod \"test-model-ea7465-kserve-65b446765-vp4km\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.712973 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.712954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:43.856347 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.856316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km"] Jul 07 00:19:43.859406 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:19:43.859380 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ff9033d_3bb9_4ef4_b5d5_740161efd279.slice/crio-01cad4446fdf1f3a2e8766cd79089f8a483171d69cffd09b24968b0e6132281a WatchSource:0}: Error finding container 01cad4446fdf1f3a2e8766cd79089f8a483171d69cffd09b24968b0e6132281a: Status 404 returned error can't find the container with id 01cad4446fdf1f3a2e8766cd79089f8a483171d69cffd09b24968b0e6132281a Jul 07 00:19:43.951740 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.951707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" event={"ID":"1ff9033d-3bb9-4ef4-b5d5-740161efd279","Type":"ContainerStarted","Data":"ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c"} Jul 07 00:19:43.951906 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.951753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" event={"ID":"1ff9033d-3bb9-4ef4-b5d5-740161efd279","Type":"ContainerStarted","Data":"01cad4446fdf1f3a2e8766cd79089f8a483171d69cffd09b24968b0e6132281a"} Jul 07 00:19:43.973031 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:43.972973 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" podStartSLOduration=0.972957377 podStartE2EDuration="972.957377ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:19:43.969839997 +0000 UTC m=+1848.620261946" watchObservedRunningTime="2026-07-07 00:19:43.972957377 +0000 UTC m=+1848.623379324" Jul 07 00:19:53.713786 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:53.713740 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:53.714185 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:53.713840 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:53.726246 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:53.726224 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:19:54.014802 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:19:54.014719 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:20:02.883753 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:02.883719 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb"] Jul 07 00:20:03.783299 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:03.783261 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km"] Jul 07 00:20:17.904798 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:17.904697 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" podUID="9186ed0c-1063-4f3e-a37d-5546814b7ea3" containerName="main" containerID="cri-o://129d1f3ae299bee8edee255372a0c550921228585ebe7ca7fcb6a2a3e1c0d151" gracePeriod=45 Jul 07 00:20:18.131577 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.131540 2564 generic.go:356] "Generic (PLEG): container finished" podID="9186ed0c-1063-4f3e-a37d-5546814b7ea3" containerID="129d1f3ae299bee8edee255372a0c550921228585ebe7ca7fcb6a2a3e1c0d151" exitCode=0 Jul 07 00:20:18.131759 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.131623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" event={"ID":"9186ed0c-1063-4f3e-a37d-5546814b7ea3","Type":"ContainerDied","Data":"129d1f3ae299bee8edee255372a0c550921228585ebe7ca7fcb6a2a3e1c0d151"} Jul 07 00:20:18.166547 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.166478 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:20:18.298547 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298491 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tls-certs\") pod \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " Jul 07 00:20:18.298721 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298576 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-model-cache\") pod \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " Jul 07 00:20:18.298721 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298593 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-dshm\") pod \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " Jul 07 00:20:18.298721 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-home\") pod \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " Jul 07 00:20:18.298721 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298643 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tmp-dir\") pod \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " Jul 07 00:20:18.298721 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298677 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mmxf\" (UniqueName: \"kubernetes.io/projected/9186ed0c-1063-4f3e-a37d-5546814b7ea3-kube-api-access-7mmxf\") pod \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\" (UID: \"9186ed0c-1063-4f3e-a37d-5546814b7ea3\") " Jul 07 00:20:18.298992 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298872 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9186ed0c-1063-4f3e-a37d-5546814b7ea3" (UID: "9186ed0c-1063-4f3e-a37d-5546814b7ea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:18.298992 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298893 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-home" (OuterVolumeSpecName: "home") pod "9186ed0c-1063-4f3e-a37d-5546814b7ea3" (UID: "9186ed0c-1063-4f3e-a37d-5546814b7ea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:18.298992 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.298939 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9186ed0c-1063-4f3e-a37d-5546814b7ea3" (UID: "9186ed0c-1063-4f3e-a37d-5546814b7ea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:18.299119 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.299013 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:18.299119 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.299032 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:18.299119 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.299044 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:18.300884 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.300863 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9186ed0c-1063-4f3e-a37d-5546814b7ea3" (UID: "9186ed0c-1063-4f3e-a37d-5546814b7ea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:20:18.300884 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.300867 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-dshm" (OuterVolumeSpecName: "dshm") pod "9186ed0c-1063-4f3e-a37d-5546814b7ea3" (UID: "9186ed0c-1063-4f3e-a37d-5546814b7ea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:18.301001 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.300927 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9186ed0c-1063-4f3e-a37d-5546814b7ea3-kube-api-access-7mmxf" (OuterVolumeSpecName: "kube-api-access-7mmxf") pod "9186ed0c-1063-4f3e-a37d-5546814b7ea3" (UID: "9186ed0c-1063-4f3e-a37d-5546814b7ea3"). InnerVolumeSpecName "kube-api-access-7mmxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:18.399876 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.399845 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9186ed0c-1063-4f3e-a37d-5546814b7ea3-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:18.399876 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.399871 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9186ed0c-1063-4f3e-a37d-5546814b7ea3-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:18.399876 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.399880 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmxf\" (UniqueName: \"kubernetes.io/projected/9186ed0c-1063-4f3e-a37d-5546814b7ea3-kube-api-access-7mmxf\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:18.804465 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:18.804406 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" podUID="1ff9033d-3bb9-4ef4-b5d5-740161efd279" containerName="main" containerID="cri-o://ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c" gracePeriod=45 Jul 07 00:20:19.082490 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.082462 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:20:19.139943 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.139912 2564 generic.go:356] "Generic (PLEG): container finished" podID="1ff9033d-3bb9-4ef4-b5d5-740161efd279" containerID="ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c" exitCode=0 Jul 07 00:20:19.140138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.139973 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" Jul 07 00:20:19.140138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.140007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" event={"ID":"1ff9033d-3bb9-4ef4-b5d5-740161efd279","Type":"ContainerDied","Data":"ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c"} Jul 07 00:20:19.140138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.140056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km" event={"ID":"1ff9033d-3bb9-4ef4-b5d5-740161efd279","Type":"ContainerDied","Data":"01cad4446fdf1f3a2e8766cd79089f8a483171d69cffd09b24968b0e6132281a"} Jul 07 00:20:19.140138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.140079 2564 scope.go:117] "RemoveContainer" containerID="ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c" Jul 07 00:20:19.141644 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.141608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" event={"ID":"9186ed0c-1063-4f3e-a37d-5546814b7ea3","Type":"ContainerDied","Data":"c580487a1654acf4095648d5a596cb3cf26f27f66ae1b418eaef35a4ef228376"} Jul 07 00:20:19.141644 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.141616 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb" Jul 07 00:20:19.149017 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.149000 2564 scope.go:117] "RemoveContainer" containerID="ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c" Jul 07 00:20:19.149360 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:20:19.149237 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c\": container with ID starting with ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c not found: ID does not exist" containerID="ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c" Jul 07 00:20:19.149360 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.149272 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c"} err="failed to get container status \"ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c\": rpc error: code = NotFound desc = could not find container \"ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c\": container with ID starting with ee10fe754dea3a07284c1457e5f8f2864be2850db989c0b48af5aa1e4921230c not found: ID does not exist" Jul 07 00:20:19.149360 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.149295 2564 scope.go:117] "RemoveContainer" containerID="129d1f3ae299bee8edee255372a0c550921228585ebe7ca7fcb6a2a3e1c0d151" Jul 07 00:20:19.162430 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.162409 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb"] Jul 07 00:20:19.167056 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.167037 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-474e44/test-model-474e44-kserve-68cc77857c-rpvgb"] Jul 07 00:20:19.207764 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.207734 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srrzs\" (UniqueName: \"kubernetes.io/projected/1ff9033d-3bb9-4ef4-b5d5-740161efd279-kube-api-access-srrzs\") pod \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " Jul 07 00:20:19.207930 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.207813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-home\") pod \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " Jul 07 00:20:19.207930 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.207859 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-dshm\") pod \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " Jul 07 00:20:19.207930 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.207874 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-model-cache\") pod \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " Jul 07 00:20:19.207930 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.207904 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tmp-dir\") pod \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " Jul 07 00:20:19.208125 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.207976 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tls-certs\") pod \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\" (UID: \"1ff9033d-3bb9-4ef4-b5d5-740161efd279\") " Jul 07 00:20:19.208183 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.208120 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ff9033d-3bb9-4ef4-b5d5-740161efd279" (UID: "1ff9033d-3bb9-4ef4-b5d5-740161efd279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:19.208183 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.208131 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-home" (OuterVolumeSpecName: "home") pod "1ff9033d-3bb9-4ef4-b5d5-740161efd279" (UID: "1ff9033d-3bb9-4ef4-b5d5-740161efd279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:19.208290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.208200 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ff9033d-3bb9-4ef4-b5d5-740161efd279" (UID: "1ff9033d-3bb9-4ef4-b5d5-740161efd279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:19.208346 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.208326 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:19.208397 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.208343 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:19.208397 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.208378 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:19.210312 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.210280 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-dshm" (OuterVolumeSpecName: "dshm") pod "1ff9033d-3bb9-4ef4-b5d5-740161efd279" (UID: "1ff9033d-3bb9-4ef4-b5d5-740161efd279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:19.210421 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.210316 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ff9033d-3bb9-4ef4-b5d5-740161efd279" (UID: "1ff9033d-3bb9-4ef4-b5d5-740161efd279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:20:19.210421 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.210329 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff9033d-3bb9-4ef4-b5d5-740161efd279-kube-api-access-srrzs" (OuterVolumeSpecName: "kube-api-access-srrzs") pod "1ff9033d-3bb9-4ef4-b5d5-740161efd279" (UID: "1ff9033d-3bb9-4ef4-b5d5-740161efd279"). InnerVolumeSpecName "kube-api-access-srrzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:19.309677 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.309640 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff9033d-3bb9-4ef4-b5d5-740161efd279-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:19.309677 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.309666 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff9033d-3bb9-4ef4-b5d5-740161efd279-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:19.309677 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.309679 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srrzs\" (UniqueName: \"kubernetes.io/projected/1ff9033d-3bb9-4ef4-b5d5-740161efd279-kube-api-access-srrzs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:19.462582 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.462548 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km"] Jul 07 00:20:19.468661 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.468626 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ea7465/test-model-ea7465-kserve-65b446765-vp4km"] Jul 07 00:20:19.806536 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.806435 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff9033d-3bb9-4ef4-b5d5-740161efd279" path="/var/lib/kubelet/pods/1ff9033d-3bb9-4ef4-b5d5-740161efd279/volumes" Jul 07 00:20:19.806896 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:19.806881 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9186ed0c-1063-4f3e-a37d-5546814b7ea3" path="/var/lib/kubelet/pods/9186ed0c-1063-4f3e-a37d-5546814b7ea3/volumes" Jul 07 00:20:53.438152 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:53.438116 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk"] Jul 07 00:20:53.438769 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:53.438381 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" containerID="cri-o://3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878" gracePeriod=30 Jul 07 00:20:58.498391 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.498343 2564 patch_prober.go:28] interesting pod/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 07 00:20:58.498859 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.498438 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 07 00:20:58.698705 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.698640 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:20:58.799795 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.799763 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-envoy\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.799795 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.799801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-token\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800012 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.799828 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn27h\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-kube-api-access-pn27h\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800012 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.799889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6f3685-c354-48b6-a996-3c3e26ac001c-istiod-ca-cert\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800012 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.799950 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-credential-socket\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800012 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.799997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-podinfo\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800220 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800042 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-data\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800220 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800082 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-socket\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800220 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800124 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-certs\") pod \"5a6f3685-c354-48b6-a996-3c3e26ac001c\" (UID: \"5a6f3685-c354-48b6-a996-3c3e26ac001c\") " Jul 07 00:20:58.800369 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800344 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:58.800432 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800374 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6f3685-c354-48b6-a996-3c3e26ac001c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:20:58.800749 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800574 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6f3685-c354-48b6-a996-3c3e26ac001c-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.800749 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800602 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.800749 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800714 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:58.800966 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.800764 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:58.801164 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.801144 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:58.802364 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.802328 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:58.802766 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.802566 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:20:58.802766 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.802648 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-kube-api-access-pn27h" (OuterVolumeSpecName: "kube-api-access-pn27h") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "kube-api-access-pn27h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:20:58.802766 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.802659 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a6f3685-c354-48b6-a996-3c3e26ac001c" (UID: "5a6f3685-c354-48b6-a996-3c3e26ac001c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:20:58.902036 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.901999 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.902036 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.902028 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.902036 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.902037 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.902036 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.902046 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.902306 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.902056 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.902306 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.902064 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn27h\" (UniqueName: \"kubernetes.io/projected/5a6f3685-c354-48b6-a996-3c3e26ac001c-kube-api-access-pn27h\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:58.902306 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:58.902073 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6f3685-c354-48b6-a996-3c3e26ac001c-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:20:59.348602 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.348572 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerID="3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878" exitCode=0 Jul 07 00:20:59.348800 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.348629 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" Jul 07 00:20:59.348800 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.348654 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" event={"ID":"5a6f3685-c354-48b6-a996-3c3e26ac001c","Type":"ContainerDied","Data":"3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878"} Jul 07 00:20:59.348800 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.348719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk" event={"ID":"5a6f3685-c354-48b6-a996-3c3e26ac001c","Type":"ContainerDied","Data":"4d45e3fff1a2001c5a1eee6af9a811685bd6a652df816b48e5027d7d8c2782c2"} Jul 07 00:20:59.348800 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.348750 2564 scope.go:117] "RemoveContainer" containerID="3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878" Jul 07 00:20:59.358349 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.358334 2564 scope.go:117] "RemoveContainer" containerID="5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d" Jul 07 00:20:59.366405 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.366388 2564 scope.go:117] "RemoveContainer" containerID="3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878" Jul 07 00:20:59.366674 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:20:59.366655 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878\": container with ID starting with 3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878 not found: ID does not exist" containerID="3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878" Jul 07 00:20:59.366743 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.366682 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878"} err="failed to get container status \"3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878\": rpc error: code = NotFound desc = could not find container \"3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878\": container with ID starting with 3bb76aa132716b54e08508924783371b61fb5183f4c1952c53519d345ab11878 not found: ID does not exist" Jul 07 00:20:59.366743 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.366699 2564 scope.go:117] "RemoveContainer" containerID="5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d" Jul 07 00:20:59.366950 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:20:59.366929 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d\": container with ID starting with 5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d not found: ID does not exist" containerID="5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d" Jul 07 00:20:59.366995 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.366959 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d"} err="failed to get container status \"5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d\": rpc error: code = NotFound desc = could not find container \"5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d\": container with ID starting with 5c2e51e4e224fc137ebe117aae55381110852a04d8bac0b22a9ae9ce6236c13d not found: ID does not exist" Jul 07 00:20:59.373251 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.373233 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk"] Jul 07 00:20:59.377247 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.377229 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ea7465-openshift-default-858bd45d7f-pgxqk"] Jul 07 00:20:59.805352 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:20:59.805320 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" path="/var/lib/kubelet/pods/5a6f3685-c354-48b6-a996-3c3e26ac001c/volumes" Jul 07 00:21:02.920190 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920157 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2"] Jul 07 00:21:02.920605 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920565 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9186ed0c-1063-4f3e-a37d-5546814b7ea3" containerName="main" Jul 07 00:21:02.920605 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920579 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9186ed0c-1063-4f3e-a37d-5546814b7ea3" containerName="main" Jul 07 00:21:02.920605 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920594 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" Jul 07 00:21:02.920605 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920600 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" Jul 07 00:21:02.920757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920608 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" Jul 07 00:21:02.920757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920615 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" Jul 07 00:21:02.920757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920632 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff9033d-3bb9-4ef4-b5d5-740161efd279" containerName="main" Jul 07 00:21:02.920757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920637 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff9033d-3bb9-4ef4-b5d5-740161efd279" containerName="main" Jul 07 00:21:02.920757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920699 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" Jul 07 00:21:02.920757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920722 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff9033d-3bb9-4ef4-b5d5-740161efd279" containerName="main" Jul 07 00:21:02.920757 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.920733 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9186ed0c-1063-4f3e-a37d-5546814b7ea3" containerName="main" Jul 07 00:21:02.925242 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.925224 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:02.927611 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.927585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-146c41-openshift-default-dockercfg-6c78r\"" Jul 07 00:21:02.945731 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:02.945703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2"] Jul 07 00:21:03.044961 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.044921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istiod-ca-cert\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045137 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.044968 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-certs\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045137 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.044994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-envoy\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045137 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.045040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-credential-socket\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045137 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.045124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-data\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045307 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.045157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2tq\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-kube-api-access-6w2tq\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045307 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.045215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-podinfo\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045307 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.045260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-socket\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.045307 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.045291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-token\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146104 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-socket\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-token\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istiod-ca-cert\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-certs\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-envoy\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-credential-socket\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146290 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-data\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146646 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2tq\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-kube-api-access-6w2tq\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146646 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-podinfo\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146646 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146529 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-socket\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146803 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146642 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-certs\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146872 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-credential-socket\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146872 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-data\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.146955 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.146919 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istiod-ca-cert\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.148949 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.148921 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-envoy\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.149125 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.149105 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-podinfo\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.156134 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.156114 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-token\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.156468 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.156448 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2tq\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-kube-api-access-6w2tq\") pod \"e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.237188 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.237100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:03.386854 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.386819 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2"] Jul 07 00:21:03.388879 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:21:03.388849 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c09e5d_1f2e_4d40_b8a9_5879cc251a1b.slice/crio-fb3812847e47f4a15d20b9941051c70b10eb727e6f7c2ae270a564bf57de6396 WatchSource:0}: Error finding container fb3812847e47f4a15d20b9941051c70b10eb727e6f7c2ae270a564bf57de6396: Status 404 returned error can't find the container with id fb3812847e47f4a15d20b9941051c70b10eb727e6f7c2ae270a564bf57de6396 Jul 07 00:21:03.391324 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.391288 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:03.391449 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.391353 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:03.391449 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:03.391382 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:04.375797 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:04.375758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" event={"ID":"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b","Type":"ContainerStarted","Data":"7c45dd709066359bd492d6cf9c648bf7b18ac2e9e16c4f57ca42f500ae65ce0f"} Jul 07 00:21:04.376197 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:04.375825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" event={"ID":"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b","Type":"ContainerStarted","Data":"fb3812847e47f4a15d20b9941051c70b10eb727e6f7c2ae270a564bf57de6396"} Jul 07 00:21:04.400917 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:04.400866 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" podStartSLOduration=2.400848417 podStartE2EDuration="2.400848417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:21:04.399436274 +0000 UTC m=+1929.049858221" watchObservedRunningTime="2026-07-07 00:21:04.400848417 +0000 UTC m=+1929.051270364" Jul 07 00:21:05.237401 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:05.237359 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:05.242693 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:05.242666 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:05.380976 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:05.380945 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:05.381988 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:05.381968 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:20.382629 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.382589 2564 patch_prober.go:28] interesting pod/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 00:21:20.383139 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.382655 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 00:21:20.479071 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.479038 2564 generic.go:356] "Generic (PLEG): container finished" podID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerID="7c45dd709066359bd492d6cf9c648bf7b18ac2e9e16c4f57ca42f500ae65ce0f" exitCode=0 Jul 07 00:21:20.479240 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.479110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" event={"ID":"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b","Type":"ContainerDied","Data":"7c45dd709066359bd492d6cf9c648bf7b18ac2e9e16c4f57ca42f500ae65ce0f"} Jul 07 00:21:20.479545 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.479530 2564 scope.go:117] "RemoveContainer" containerID="7c45dd709066359bd492d6cf9c648bf7b18ac2e9e16c4f57ca42f500ae65ce0f" Jul 07 00:21:20.480603 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.480577 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:20.480674 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.480645 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:20.480711 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:20.480676 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:21.486234 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:21.486198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" event={"ID":"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b","Type":"ContainerStarted","Data":"7099d7deb611d78fe23d305cc916730ca74b78a93585563c655f53829c98f72e"} Jul 07 00:21:22.237768 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:22.237720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:22.238226 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:22.238194 2564 patch_prober.go:28] interesting pod/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 00:21:22.238323 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:22.238281 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 00:21:23.238580 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:23.238544 2564 patch_prober.go:28] interesting pod/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 00:21:23.239157 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:23.239128 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 00:21:24.238558 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:24.238519 2564 patch_prober.go:28] interesting pod/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 00:21:24.238804 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:24.238603 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 00:21:25.238230 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:25.238182 2564 patch_prober.go:28] interesting pod/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 00:21:25.238564 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:25.238255 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 00:21:26.242632 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:26.242605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:26.243120 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:26.242991 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:26.243969 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:26.243949 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:32.398515 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:32.398468 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2"] Jul 07 00:21:32.398945 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:32.398702 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" containerID="cri-o://7099d7deb611d78fe23d305cc916730ca74b78a93585563c655f53829c98f72e" gracePeriod=30 Jul 07 00:21:37.580322 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.580292 2564 generic.go:356] "Generic (PLEG): container finished" podID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerID="7099d7deb611d78fe23d305cc916730ca74b78a93585563c655f53829c98f72e" exitCode=0 Jul 07 00:21:37.580718 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.580361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" event={"ID":"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b","Type":"ContainerDied","Data":"7099d7deb611d78fe23d305cc916730ca74b78a93585563c655f53829c98f72e"} Jul 07 00:21:37.580718 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.580426 2564 scope.go:117] "RemoveContainer" containerID="7c45dd709066359bd492d6cf9c648bf7b18ac2e9e16c4f57ca42f500ae65ce0f" Jul 07 00:21:37.663728 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.663674 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:37.807022 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.806999 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-podinfo\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807202 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-envoy\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807202 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807083 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istiod-ca-cert\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807202 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807105 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w2tq\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-kube-api-access-6w2tq\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807202 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807133 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-token\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807202 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807181 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-certs\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807477 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807223 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-credential-socket\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807477 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807279 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-data\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807477 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807303 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-socket\") pod \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\" (UID: \"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b\") " Jul 07 00:21:37.807638 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807537 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:37.807638 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807553 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:37.807638 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807547 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:21:37.807794 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807707 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.807794 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807731 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.807794 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807730 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:37.807794 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.807743 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.808232 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.808212 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:37.809559 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.809531 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:21:37.809559 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.809545 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:21:37.809773 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.809720 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:21:37.809840 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.809822 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-kube-api-access-6w2tq" (OuterVolumeSpecName: "kube-api-access-6w2tq") pod "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" (UID: "64c09e5d-1f2e-4d40-b8a9-5879cc251a1b"). InnerVolumeSpecName "kube-api-access-6w2tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:21:37.908772 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.908738 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2tq\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-kube-api-access-6w2tq\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.908772 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.908767 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.908772 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.908778 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.908996 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.908785 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.908996 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.908793 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:37.908996 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:37.908801 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:21:38.586359 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:38.586319 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" Jul 07 00:21:38.586359 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:38.586349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2" event={"ID":"64c09e5d-1f2e-4d40-b8a9-5879cc251a1b","Type":"ContainerDied","Data":"fb3812847e47f4a15d20b9941051c70b10eb727e6f7c2ae270a564bf57de6396"} Jul 07 00:21:38.586873 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:38.586404 2564 scope.go:117] "RemoveContainer" containerID="7099d7deb611d78fe23d305cc916730ca74b78a93585563c655f53829c98f72e" Jul 07 00:21:38.607237 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:38.607216 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2"] Jul 07 00:21:38.611075 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:38.611056 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-146c41-openshift-default-f578d4b67-h4fd2"] Jul 07 00:21:39.806371 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:39.806336 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" path="/var/lib/kubelet/pods/64c09e5d-1f2e-4d40-b8a9-5879cc251a1b/volumes" Jul 07 00:21:40.930975 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.930945 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g"] Jul 07 00:21:40.931336 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.931324 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" Jul 07 00:21:40.931380 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.931337 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" Jul 07 00:21:40.931380 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.931347 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" Jul 07 00:21:40.931380 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.931352 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" Jul 07 00:21:40.931472 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.931424 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" Jul 07 00:21:40.931472 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.931432 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6f3685-c354-48b6-a996-3c3e26ac001c" containerName="istio-proxy" Jul 07 00:21:40.931472 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.931440 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c09e5d-1f2e-4d40-b8a9-5879cc251a1b" containerName="istio-proxy" Jul 07 00:21:40.936217 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.936199 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:40.938861 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.938832 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-26a9dd-openshift-default-dockercfg-7t5rj\"" Jul 07 00:21:40.952882 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:40.952859 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g"] Jul 07 00:21:41.036343 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-podinfo\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036567 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-envoy\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036567 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-certs\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036567 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-socket\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036700 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-token\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036750 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-data\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036750 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istiod-ca-cert\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036903 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zbj\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-kube-api-access-c8zbj\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.036903 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.036814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-credential-socket\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138396 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-token\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138610 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-data\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138610 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istiod-ca-cert\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138610 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zbj\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-kube-api-access-c8zbj\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138610 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-credential-socket\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138610 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-podinfo\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138893 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-envoy\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138893 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-certs\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.138893 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-socket\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.139074 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.138983 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-credential-socket\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.139138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.139117 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-socket\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.139241 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.139218 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-certs\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.139765 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.139729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-data\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.139765 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.139748 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istiod-ca-cert\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.141356 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.141330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-envoy\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.141859 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.141840 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-podinfo\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.147545 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.147518 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zbj\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-kube-api-access-c8zbj\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.147999 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.147983 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-token\") pod \"e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.247666 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.247572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:41.395638 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.395613 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g"] Jul 07 00:21:41.397605 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:21:41.397570 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8cfa35_10ec_4b88_858d_37fa4ecac12e.slice/crio-bf67f26456c4966d1dd5b331197453e65717bc3c7e695a564c92e3a6343ec4ae WatchSource:0}: Error finding container bf67f26456c4966d1dd5b331197453e65717bc3c7e695a564c92e3a6343ec4ae: Status 404 returned error can't find the container with id bf67f26456c4966d1dd5b331197453e65717bc3c7e695a564c92e3a6343ec4ae Jul 07 00:21:41.399824 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.399793 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:41.399931 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.399856 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:41.399931 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.399883 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:41.605674 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.605574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" event={"ID":"7c8cfa35-10ec-4b88-858d-37fa4ecac12e","Type":"ContainerStarted","Data":"d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769"} Jul 07 00:21:41.605674 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.605640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" event={"ID":"7c8cfa35-10ec-4b88-858d-37fa4ecac12e","Type":"ContainerStarted","Data":"bf67f26456c4966d1dd5b331197453e65717bc3c7e695a564c92e3a6343ec4ae"} Jul 07 00:21:41.627209 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:41.627135 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" podStartSLOduration=1.62711549 podStartE2EDuration="1.62711549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:21:41.624234766 +0000 UTC m=+1966.274656740" watchObservedRunningTime="2026-07-07 00:21:41.62711549 +0000 UTC m=+1966.277537438" Jul 07 00:21:42.248707 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:42.248660 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:42.254089 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:42.254062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:42.610813 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:42.610735 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:42.611712 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:42.611695 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:55.691410 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:55.691378 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerID="d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769" exitCode=0 Jul 07 00:21:55.691873 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:55.691443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" event={"ID":"7c8cfa35-10ec-4b88-858d-37fa4ecac12e","Type":"ContainerDied","Data":"d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769"} Jul 07 00:21:55.691873 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:55.691849 2564 scope.go:117] "RemoveContainer" containerID="d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769" Jul 07 00:21:55.692995 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:55.692957 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:55.693120 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:55.693026 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:55.693120 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:55.693057 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 07 00:21:56.698238 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:56.698197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" event={"ID":"7c8cfa35-10ec-4b88-858d-37fa4ecac12e","Type":"ContainerStarted","Data":"504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01"} Jul 07 00:21:57.248221 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:57.248124 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:57.249515 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:57.249467 2564 patch_prober.go:28] interesting pod/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:21:57.249641 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:57.249549 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 07 00:21:57.611707 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:57.611621 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:21:58.248689 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:58.248647 2564 patch_prober.go:28] interesting pod/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:21:58.249157 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:58.248739 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 07 00:21:59.248511 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:59.248448 2564 patch_prober.go:28] interesting pod/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:21:59.248711 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:21:59.248544 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 07 00:22:00.248581 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:00.248087 2564 patch_prober.go:28] interesting pod/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 07 00:22:00.248581 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:00.248151 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 07 00:22:01.252115 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:01.252083 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:22:01.253142 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:01.253122 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:22:10.595253 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.595219 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm"] Jul 07 00:22:10.599411 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.599391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.601584 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.601560 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"openshift-service-ca.crt\"" Jul 07 00:22:10.601703 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.601584 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"default-dockercfg-cfq69\"" Jul 07 00:22:10.602188 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.602169 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"kube-root-ca.crt\"" Jul 07 00:22:10.602317 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.602245 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"auth-test-model-a7bd37-kserve-self-signed-certs\"" Jul 07 00:22:10.609282 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.609261 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm"] Jul 07 00:22:10.716836 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.716800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-home\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.717023 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.716895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88k4x\" (UniqueName: \"kubernetes.io/projected/e60f582d-5594-4a1a-8330-b2fb229bc534-kube-api-access-88k4x\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.717165 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.717022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60f582d-5594-4a1a-8330-b2fb229bc534-tls-certs\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.717165 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.717066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-dshm\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.717165 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.717094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-tmp-dir\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.717165 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.717120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-model-cache\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818241 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-dshm\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818241 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-tmp-dir\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-model-cache\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-home\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88k4x\" (UniqueName: \"kubernetes.io/projected/e60f582d-5594-4a1a-8330-b2fb229bc534-kube-api-access-88k4x\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818437 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60f582d-5594-4a1a-8330-b2fb229bc534-tls-certs\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818863 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818717 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-tmp-dir\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818863 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818769 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-model-cache\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.818863 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.818787 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-home\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.820663 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.820648 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-dshm\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.820920 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.820904 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60f582d-5594-4a1a-8330-b2fb229bc534-tls-certs\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.826544 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.826522 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88k4x\" (UniqueName: \"kubernetes.io/projected/e60f582d-5594-4a1a-8330-b2fb229bc534-kube-api-access-88k4x\") pod \"auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:10.923573 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:10.923541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:11.067828 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:11.067803 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm"] Jul 07 00:22:11.070436 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:22:11.070408 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode60f582d_5594_4a1a_8330_b2fb229bc534.slice/crio-3f007728f35c1876406145314cf494117ca488d40347873704f18c6a2ef91bf1 WatchSource:0}: Error finding container 3f007728f35c1876406145314cf494117ca488d40347873704f18c6a2ef91bf1: Status 404 returned error can't find the container with id 3f007728f35c1876406145314cf494117ca488d40347873704f18c6a2ef91bf1 Jul 07 00:22:11.783359 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:11.783320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" event={"ID":"e60f582d-5594-4a1a-8330-b2fb229bc534","Type":"ContainerStarted","Data":"d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451"} Jul 07 00:22:11.783851 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:11.783366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" event={"ID":"e60f582d-5594-4a1a-8330-b2fb229bc534","Type":"ContainerStarted","Data":"3f007728f35c1876406145314cf494117ca488d40347873704f18c6a2ef91bf1"} Jul 07 00:22:11.803409 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:11.803359 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" podStartSLOduration=1.803341066 podStartE2EDuration="1.803341066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:22:11.801118789 +0000 UTC m=+1996.451540747" watchObservedRunningTime="2026-07-07 00:22:11.803341066 +0000 UTC m=+1996.453763057" Jul 07 00:22:20.924504 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:20.924470 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:20.924973 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:20.924545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:20.939600 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:20.939567 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:21.845927 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:21.845898 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:22:30.293827 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.293781 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z"] Jul 07 00:22:30.298326 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.298294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.301935 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.301718 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"kube-root-ca.crt\"" Jul 07 00:22:30.301935 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.301843 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"default-dockercfg-skmrn\"" Jul 07 00:22:30.301935 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.301863 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"openshift-service-ca.crt\"" Jul 07 00:22:30.302209 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.301997 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"auth-test-model-26a9dd-kserve-self-signed-certs\"" Jul 07 00:22:30.308277 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.308248 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z"] Jul 07 00:22:30.405522 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.405464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-dshm\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.405688 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.405541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-home\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.405688 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.405604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-model-cache\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.405688 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.405641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqjm\" (UniqueName: \"kubernetes.io/projected/ef0b4887-8e70-4d58-be23-ed5e78befbc9-kube-api-access-fdqjm\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.405688 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.405668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tls-certs\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.405827 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.405790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tmp-dir\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.506843 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.506809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tls-certs\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507022 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.506868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tmp-dir\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507022 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.506921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-dshm\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507022 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.506963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-home\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507125 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.507039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-model-cache\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507125 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.507090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqjm\" (UniqueName: \"kubernetes.io/projected/ef0b4887-8e70-4d58-be23-ed5e78befbc9-kube-api-access-fdqjm\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507352 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.507330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tmp-dir\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507417 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.507369 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-home\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.507545 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.507523 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-model-cache\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.509425 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.509405 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-dshm\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.509556 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.509539 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tls-certs\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.516947 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.516927 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqjm\" (UniqueName: \"kubernetes.io/projected/ef0b4887-8e70-4d58-be23-ed5e78befbc9-kube-api-access-fdqjm\") pod \"auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.610734 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.610658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:30.749772 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.749748 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z"] Jul 07 00:22:30.752761 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:22:30.752718 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef0b4887_8e70_4d58_be23_ed5e78befbc9.slice/crio-70b5566e9eb3d2adf57eb245d9f84016478b741a0b4a3b52a35582e52fe95360 WatchSource:0}: Error finding container 70b5566e9eb3d2adf57eb245d9f84016478b741a0b4a3b52a35582e52fe95360: Status 404 returned error can't find the container with id 70b5566e9eb3d2adf57eb245d9f84016478b741a0b4a3b52a35582e52fe95360 Jul 07 00:22:30.884189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.884150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" event={"ID":"ef0b4887-8e70-4d58-be23-ed5e78befbc9","Type":"ContainerStarted","Data":"b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b"} Jul 07 00:22:30.884359 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.884197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" event={"ID":"ef0b4887-8e70-4d58-be23-ed5e78befbc9","Type":"ContainerStarted","Data":"70b5566e9eb3d2adf57eb245d9f84016478b741a0b4a3b52a35582e52fe95360"} Jul 07 00:22:30.902085 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:30.902041 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" podStartSLOduration=0.902028118 podStartE2EDuration="902.028118ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:22:30.901718403 +0000 UTC m=+2015.552140347" watchObservedRunningTime="2026-07-07 00:22:30.902028118 +0000 UTC m=+2015.552450065" Jul 07 00:22:40.611088 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:40.611051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:40.611592 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:40.611106 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:40.623587 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:40.623560 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:22:40.943052 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:22:40.943019 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:23:55.892532 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:23:55.892451 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:23:55.896063 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:23:55.896041 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:24:26.180385 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:26.180348 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm"] Jul 07 00:24:27.093995 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:27.093964 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z"] Jul 07 00:24:41.200825 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.200764 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" podUID="e60f582d-5594-4a1a-8330-b2fb229bc534" containerName="main" containerID="cri-o://d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451" gracePeriod=45 Jul 07 00:24:41.466789 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.466766 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:24:41.561087 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.561054 2564 generic.go:356] "Generic (PLEG): container finished" podID="e60f582d-5594-4a1a-8330-b2fb229bc534" containerID="d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451" exitCode=0 Jul 07 00:24:41.561347 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.561104 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" Jul 07 00:24:41.561347 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.561137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" event={"ID":"e60f582d-5594-4a1a-8330-b2fb229bc534","Type":"ContainerDied","Data":"d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451"} Jul 07 00:24:41.561347 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.561182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm" event={"ID":"e60f582d-5594-4a1a-8330-b2fb229bc534","Type":"ContainerDied","Data":"3f007728f35c1876406145314cf494117ca488d40347873704f18c6a2ef91bf1"} Jul 07 00:24:41.561347 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.561199 2564 scope.go:117] "RemoveContainer" containerID="d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451" Jul 07 00:24:41.570171 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.570154 2564 scope.go:117] "RemoveContainer" containerID="d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451" Jul 07 00:24:41.570421 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:24:41.570404 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451\": container with ID starting with d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451 not found: ID does not exist" containerID="d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451" Jul 07 00:24:41.570480 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.570432 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451"} err="failed to get container status \"d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451\": rpc error: code = NotFound desc = could not find container \"d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451\": container with ID starting with d9169f8991824e6dbe0004e51f2c45ac0175b8be750b4f34aaa2696d45dcd451 not found: ID does not exist" Jul 07 00:24:41.595768 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.595743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-home\") pod \"e60f582d-5594-4a1a-8330-b2fb229bc534\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " Jul 07 00:24:41.595875 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.595789 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-dshm\") pod \"e60f582d-5594-4a1a-8330-b2fb229bc534\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " Jul 07 00:24:41.595875 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.595827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-model-cache\") pod \"e60f582d-5594-4a1a-8330-b2fb229bc534\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " Jul 07 00:24:41.595875 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.595850 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60f582d-5594-4a1a-8330-b2fb229bc534-tls-certs\") pod \"e60f582d-5594-4a1a-8330-b2fb229bc534\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " Jul 07 00:24:41.596038 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.595896 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88k4x\" (UniqueName: \"kubernetes.io/projected/e60f582d-5594-4a1a-8330-b2fb229bc534-kube-api-access-88k4x\") pod \"e60f582d-5594-4a1a-8330-b2fb229bc534\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " Jul 07 00:24:41.596038 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.595915 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-tmp-dir\") pod \"e60f582d-5594-4a1a-8330-b2fb229bc534\" (UID: \"e60f582d-5594-4a1a-8330-b2fb229bc534\") " Jul 07 00:24:41.596140 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.596055 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-home" (OuterVolumeSpecName: "home") pod "e60f582d-5594-4a1a-8330-b2fb229bc534" (UID: "e60f582d-5594-4a1a-8330-b2fb229bc534"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:41.596140 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.596120 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-model-cache" (OuterVolumeSpecName: "model-cache") pod "e60f582d-5594-4a1a-8330-b2fb229bc534" (UID: "e60f582d-5594-4a1a-8330-b2fb229bc534"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:41.596254 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.596226 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e60f582d-5594-4a1a-8330-b2fb229bc534" (UID: "e60f582d-5594-4a1a-8330-b2fb229bc534"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:41.596403 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.596389 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:41.596468 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.596406 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:41.596468 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.596414 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:41.598159 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.598139 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-dshm" (OuterVolumeSpecName: "dshm") pod "e60f582d-5594-4a1a-8330-b2fb229bc534" (UID: "e60f582d-5594-4a1a-8330-b2fb229bc534"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:41.598159 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.598145 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60f582d-5594-4a1a-8330-b2fb229bc534-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e60f582d-5594-4a1a-8330-b2fb229bc534" (UID: "e60f582d-5594-4a1a-8330-b2fb229bc534"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:24:41.598275 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.598201 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60f582d-5594-4a1a-8330-b2fb229bc534-kube-api-access-88k4x" (OuterVolumeSpecName: "kube-api-access-88k4x") pod "e60f582d-5594-4a1a-8330-b2fb229bc534" (UID: "e60f582d-5594-4a1a-8330-b2fb229bc534"). InnerVolumeSpecName "kube-api-access-88k4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:24:41.696906 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.696877 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60f582d-5594-4a1a-8330-b2fb229bc534-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:41.696906 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.696904 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88k4x\" (UniqueName: \"kubernetes.io/projected/e60f582d-5594-4a1a-8330-b2fb229bc534-kube-api-access-88k4x\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:41.697107 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.696914 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60f582d-5594-4a1a-8330-b2fb229bc534-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:41.875948 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.875915 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm"] Jul 07 00:24:41.879390 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:41.879370 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/auth-test-model-a7bd37-kserve-757b6bbb99-4zrjm"] Jul 07 00:24:42.114800 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.114714 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" podUID="ef0b4887-8e70-4d58-be23-ed5e78befbc9" containerName="main" containerID="cri-o://b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b" gracePeriod=45 Jul 07 00:24:42.373860 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.373802 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:24:42.504755 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.504728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-home\") pod \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " Jul 07 00:24:42.504924 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.504791 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tls-certs\") pod \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " Jul 07 00:24:42.504924 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.504820 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-model-cache\") pod \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " Jul 07 00:24:42.504924 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.504866 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-dshm\") pod \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " Jul 07 00:24:42.504924 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.504899 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdqjm\" (UniqueName: \"kubernetes.io/projected/ef0b4887-8e70-4d58-be23-ed5e78befbc9-kube-api-access-fdqjm\") pod \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " Jul 07 00:24:42.505139 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.504934 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tmp-dir\") pod \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\" (UID: \"ef0b4887-8e70-4d58-be23-ed5e78befbc9\") " Jul 07 00:24:42.505139 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.505045 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-home" (OuterVolumeSpecName: "home") pod "ef0b4887-8e70-4d58-be23-ed5e78befbc9" (UID: "ef0b4887-8e70-4d58-be23-ed5e78befbc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:42.505139 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.505087 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef0b4887-8e70-4d58-be23-ed5e78befbc9" (UID: "ef0b4887-8e70-4d58-be23-ed5e78befbc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:42.505294 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.505214 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef0b4887-8e70-4d58-be23-ed5e78befbc9" (UID: "ef0b4887-8e70-4d58-be23-ed5e78befbc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:42.505346 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.505304 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:42.505346 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.505318 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:42.505346 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.505326 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:42.507069 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.507050 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-dshm" (OuterVolumeSpecName: "dshm") pod "ef0b4887-8e70-4d58-be23-ed5e78befbc9" (UID: "ef0b4887-8e70-4d58-be23-ed5e78befbc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:24:42.507184 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.507083 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef0b4887-8e70-4d58-be23-ed5e78befbc9" (UID: "ef0b4887-8e70-4d58-be23-ed5e78befbc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:24:42.507184 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.507169 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0b4887-8e70-4d58-be23-ed5e78befbc9-kube-api-access-fdqjm" (OuterVolumeSpecName: "kube-api-access-fdqjm") pod "ef0b4887-8e70-4d58-be23-ed5e78befbc9" (UID: "ef0b4887-8e70-4d58-be23-ed5e78befbc9"). InnerVolumeSpecName "kube-api-access-fdqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:24:42.567342 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.567311 2564 generic.go:356] "Generic (PLEG): container finished" podID="ef0b4887-8e70-4d58-be23-ed5e78befbc9" containerID="b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b" exitCode=0 Jul 07 00:24:42.567529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.567355 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" Jul 07 00:24:42.567529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.567406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" event={"ID":"ef0b4887-8e70-4d58-be23-ed5e78befbc9","Type":"ContainerDied","Data":"b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b"} Jul 07 00:24:42.567529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.567449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z" event={"ID":"ef0b4887-8e70-4d58-be23-ed5e78befbc9","Type":"ContainerDied","Data":"70b5566e9eb3d2adf57eb245d9f84016478b741a0b4a3b52a35582e52fe95360"} Jul 07 00:24:42.567529 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.567472 2564 scope.go:117] "RemoveContainer" containerID="b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b" Jul 07 00:24:42.577297 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.577279 2564 scope.go:117] "RemoveContainer" containerID="b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b" Jul 07 00:24:42.577597 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:24:42.577572 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b\": container with ID starting with b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b not found: ID does not exist" containerID="b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b" Jul 07 00:24:42.577650 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.577599 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b"} err="failed to get container status \"b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b\": rpc error: code = NotFound desc = could not find container \"b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b\": container with ID starting with b397b0b593f8baff065085e12012d1480f6b167fc85d638082d3828dcacd4a6b not found: ID does not exist" Jul 07 00:24:42.588358 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.588335 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z"] Jul 07 00:24:42.592753 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.592735 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/auth-test-model-26a9dd-kserve-5c4cc5d6c5-fxq9z"] Jul 07 00:24:42.606343 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.606327 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0b4887-8e70-4d58-be23-ed5e78befbc9-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:42.606409 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.606345 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef0b4887-8e70-4d58-be23-ed5e78befbc9-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:42.606409 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:42.606354 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqjm\" (UniqueName: \"kubernetes.io/projected/ef0b4887-8e70-4d58-be23-ed5e78befbc9-kube-api-access-fdqjm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:24:43.805731 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:43.805688 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60f582d-5594-4a1a-8330-b2fb229bc534" path="/var/lib/kubelet/pods/e60f582d-5594-4a1a-8330-b2fb229bc534/volumes" Jul 07 00:24:43.806277 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:24:43.806256 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef0b4887-8e70-4d58-be23-ed5e78befbc9" path="/var/lib/kubelet/pods/ef0b4887-8e70-4d58-be23-ed5e78befbc9/volumes" Jul 07 00:25:19.996457 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:19.996415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft"] Jul 07 00:25:19.997011 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:19.996868 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60f582d-5594-4a1a-8330-b2fb229bc534" containerName="main" Jul 07 00:25:19.997011 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:19.996882 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60f582d-5594-4a1a-8330-b2fb229bc534" containerName="main" Jul 07 00:25:19.997011 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:19.996917 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef0b4887-8e70-4d58-be23-ed5e78befbc9" containerName="main" Jul 07 00:25:19.997011 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:19.996927 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef0b4887-8e70-4d58-be23-ed5e78befbc9" containerName="main" Jul 07 00:25:19.997249 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:19.997020 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60f582d-5594-4a1a-8330-b2fb229bc534" containerName="main" Jul 07 00:25:19.997249 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:19.997036 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef0b4887-8e70-4d58-be23-ed5e78befbc9" containerName="main" Jul 07 00:25:20.000550 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.000531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.003572 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.003550 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"default-dockercfg-skmrn\"" Jul 07 00:25:20.003667 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.003583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"sub-test-model-26a9dd-kserve-self-signed-certs\"" Jul 07 00:25:20.003667 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.003603 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"kube-root-ca.crt\"" Jul 07 00:25:20.003667 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.003552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"openshift-service-ca.crt\"" Jul 07 00:25:20.009604 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.009585 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft"] Jul 07 00:25:20.056197 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.056158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-home\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.056351 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.056207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-tmp-dir\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.056351 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.056294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1533e90b-5e64-4eda-b529-a11fc102b586-tls-certs\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.056351 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.056323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwkh\" (UniqueName: \"kubernetes.io/projected/1533e90b-5e64-4eda-b529-a11fc102b586-kube-api-access-prwkh\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.056554 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.056355 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-dshm\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.056554 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.056387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-model-cache\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.157935 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.157898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-dshm\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158106 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.157951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-model-cache\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158106 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.158007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-home\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158106 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.158039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-tmp-dir\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158106 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.158077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1533e90b-5e64-4eda-b529-a11fc102b586-tls-certs\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158106 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.158099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwkh\" (UniqueName: \"kubernetes.io/projected/1533e90b-5e64-4eda-b529-a11fc102b586-kube-api-access-prwkh\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158425 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.158392 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-model-cache\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158568 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.158447 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-tmp-dir\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.158568 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.158479 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-home\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.160247 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.160226 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-dshm\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.160706 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.160687 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1533e90b-5e64-4eda-b529-a11fc102b586-tls-certs\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.165369 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.165347 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwkh\" (UniqueName: \"kubernetes.io/projected/1533e90b-5e64-4eda-b529-a11fc102b586-kube-api-access-prwkh\") pod \"sub-test-model-26a9dd-kserve-5b4556c95-plsft\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.322962 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.322881 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:20.460809 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.460787 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft"] Jul 07 00:25:20.462767 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:25:20.462736 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1533e90b_5e64_4eda_b529_a11fc102b586.slice/crio-dd13ca79ae76ac7094b3a7b9e73cad5fc1be47278a19099a093c536b6d86d635 WatchSource:0}: Error finding container dd13ca79ae76ac7094b3a7b9e73cad5fc1be47278a19099a093c536b6d86d635: Status 404 returned error can't find the container with id dd13ca79ae76ac7094b3a7b9e73cad5fc1be47278a19099a093c536b6d86d635 Jul 07 00:25:20.764335 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.764295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" event={"ID":"1533e90b-5e64-4eda-b529-a11fc102b586","Type":"ContainerStarted","Data":"b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9"} Jul 07 00:25:20.764523 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.764346 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" event={"ID":"1533e90b-5e64-4eda-b529-a11fc102b586","Type":"ContainerStarted","Data":"dd13ca79ae76ac7094b3a7b9e73cad5fc1be47278a19099a093c536b6d86d635"} Jul 07 00:25:20.780285 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:20.780238 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" podStartSLOduration=1.7802240729999999 podStartE2EDuration="1.780224073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:25:20.778383048 +0000 UTC m=+2185.428804995" watchObservedRunningTime="2026-07-07 00:25:20.780224073 +0000 UTC m=+2185.430646020" Jul 07 00:25:30.323388 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:30.323345 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:30.323934 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:30.323434 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:30.335956 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:30.335932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:30.825483 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:30.825456 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:25:58.984299 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:25:58.984266 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft"] Jul 07 00:26:14.005159 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.005087 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" podUID="1533e90b-5e64-4eda-b529-a11fc102b586" containerName="main" containerID="cri-o://b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9" gracePeriod=45 Jul 07 00:26:14.263883 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.263826 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:26:14.285605 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.285576 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-dshm\") pod \"1533e90b-5e64-4eda-b529-a11fc102b586\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " Jul 07 00:26:14.285753 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.285619 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-home\") pod \"1533e90b-5e64-4eda-b529-a11fc102b586\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " Jul 07 00:26:14.285753 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.285734 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1533e90b-5e64-4eda-b529-a11fc102b586-tls-certs\") pod \"1533e90b-5e64-4eda-b529-a11fc102b586\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " Jul 07 00:26:14.285880 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.285761 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-tmp-dir\") pod \"1533e90b-5e64-4eda-b529-a11fc102b586\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " Jul 07 00:26:14.285880 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.285819 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prwkh\" (UniqueName: \"kubernetes.io/projected/1533e90b-5e64-4eda-b529-a11fc102b586-kube-api-access-prwkh\") pod \"1533e90b-5e64-4eda-b529-a11fc102b586\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " Jul 07 00:26:14.285880 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.285862 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-model-cache\") pod \"1533e90b-5e64-4eda-b529-a11fc102b586\" (UID: \"1533e90b-5e64-4eda-b529-a11fc102b586\") " Jul 07 00:26:14.286048 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.285973 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-home" (OuterVolumeSpecName: "home") pod "1533e90b-5e64-4eda-b529-a11fc102b586" (UID: "1533e90b-5e64-4eda-b529-a11fc102b586"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:14.286280 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.286190 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-model-cache" (OuterVolumeSpecName: "model-cache") pod "1533e90b-5e64-4eda-b529-a11fc102b586" (UID: "1533e90b-5e64-4eda-b529-a11fc102b586"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:14.286592 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.286311 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1533e90b-5e64-4eda-b529-a11fc102b586" (UID: "1533e90b-5e64-4eda-b529-a11fc102b586"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:14.286766 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.286743 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:26:14.286878 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.286772 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:26:14.288304 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.288282 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-dshm" (OuterVolumeSpecName: "dshm") pod "1533e90b-5e64-4eda-b529-a11fc102b586" (UID: "1533e90b-5e64-4eda-b529-a11fc102b586"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:26:14.288618 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.288528 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1533e90b-5e64-4eda-b529-a11fc102b586-kube-api-access-prwkh" (OuterVolumeSpecName: "kube-api-access-prwkh") pod "1533e90b-5e64-4eda-b529-a11fc102b586" (UID: "1533e90b-5e64-4eda-b529-a11fc102b586"). InnerVolumeSpecName "kube-api-access-prwkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:26:14.288618 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.288597 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1533e90b-5e64-4eda-b529-a11fc102b586-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1533e90b-5e64-4eda-b529-a11fc102b586" (UID: "1533e90b-5e64-4eda-b529-a11fc102b586"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:26:14.388007 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.387980 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prwkh\" (UniqueName: \"kubernetes.io/projected/1533e90b-5e64-4eda-b529-a11fc102b586-kube-api-access-prwkh\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:26:14.388007 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.388005 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:26:14.388192 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.388016 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1533e90b-5e64-4eda-b529-a11fc102b586-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:26:14.388192 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.388026 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1533e90b-5e64-4eda-b529-a11fc102b586-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:26:14.699008 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.698976 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9"] Jul 07 00:26:14.699627 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.699565 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1533e90b-5e64-4eda-b529-a11fc102b586" containerName="main" Jul 07 00:26:14.699627 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.699586 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1533e90b-5e64-4eda-b529-a11fc102b586" containerName="main" Jul 07 00:26:14.699818 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.699698 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1533e90b-5e64-4eda-b529-a11fc102b586" containerName="main" Jul 07 00:26:14.702987 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.702969 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.705301 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.705278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"kube-root-ca.crt\"" Jul 07 00:26:14.705846 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.705819 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"rate-test-model-a7bd37-kserve-self-signed-certs\"" Jul 07 00:26:14.705950 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.705856 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"openshift-service-ca.crt\"" Jul 07 00:26:14.705950 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.705897 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7bd37\"/\"default-dockercfg-cfq69\"" Jul 07 00:26:14.713945 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.713920 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9"] Jul 07 00:26:14.791704 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.791669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-tmp-dir\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.791867 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.791716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-model-cache\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.791867 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.791732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-dshm\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.791867 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.791831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-home\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.791976 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.791879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffdc8\" (UniqueName: \"kubernetes.io/projected/a09f5c41-7ca1-4333-9339-433273efa70b-kube-api-access-ffdc8\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.791976 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.791900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f5c41-7ca1-4333-9339-433273efa70b-tls-certs\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893270 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-home\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893462 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffdc8\" (UniqueName: \"kubernetes.io/projected/a09f5c41-7ca1-4333-9339-433273efa70b-kube-api-access-ffdc8\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893462 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f5c41-7ca1-4333-9339-433273efa70b-tls-certs\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893462 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-tmp-dir\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893462 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893414 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-model-cache\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893462 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-dshm\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893769 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893671 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-home\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893825 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893791 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-tmp-dir\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.893874 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.893835 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-model-cache\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.895784 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.895763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-dshm\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.896073 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.896054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f5c41-7ca1-4333-9339-433273efa70b-tls-certs\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:14.901823 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:14.901802 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffdc8\" (UniqueName: \"kubernetes.io/projected/a09f5c41-7ca1-4333-9339-433273efa70b-kube-api-access-ffdc8\") pod \"rate-test-model-a7bd37-kserve-867bb84bf5-z86s9\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:15.014559 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.014471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:15.039471 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.039443 2564 generic.go:356] "Generic (PLEG): container finished" podID="1533e90b-5e64-4eda-b529-a11fc102b586" containerID="b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9" exitCode=0 Jul 07 00:26:15.039632 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.039522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" event={"ID":"1533e90b-5e64-4eda-b529-a11fc102b586","Type":"ContainerDied","Data":"b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9"} Jul 07 00:26:15.039632 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.039538 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" Jul 07 00:26:15.039632 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.039558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft" event={"ID":"1533e90b-5e64-4eda-b529-a11fc102b586","Type":"ContainerDied","Data":"dd13ca79ae76ac7094b3a7b9e73cad5fc1be47278a19099a093c536b6d86d635"} Jul 07 00:26:15.039632 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.039576 2564 scope.go:117] "RemoveContainer" containerID="b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9" Jul 07 00:26:15.052477 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.052448 2564 scope.go:117] "RemoveContainer" containerID="b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9" Jul 07 00:26:15.052894 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:26:15.052868 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9\": container with ID starting with b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9 not found: ID does not exist" containerID="b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9" Jul 07 00:26:15.052988 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.052904 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9"} err="failed to get container status \"b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9\": rpc error: code = NotFound desc = could not find container \"b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9\": container with ID starting with b904a5c4657c2d24328d67010f1e5f0d76c1681425b3e27e34b70ae65fac1ee9 not found: ID does not exist" Jul 07 00:26:15.067280 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.067209 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft"] Jul 07 00:26:15.074911 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.074884 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/sub-test-model-26a9dd-kserve-5b4556c95-plsft"] Jul 07 00:26:15.156185 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.156161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9"] Jul 07 00:26:15.158078 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:26:15.158054 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda09f5c41_7ca1_4333_9339_433273efa70b.slice/crio-e8f0b1ee5295a6a1dae4e45576814057af11e2e13da5419992573f4eb0e1f2b3 WatchSource:0}: Error finding container e8f0b1ee5295a6a1dae4e45576814057af11e2e13da5419992573f4eb0e1f2b3: Status 404 returned error can't find the container with id e8f0b1ee5295a6a1dae4e45576814057af11e2e13da5419992573f4eb0e1f2b3 Jul 07 00:26:15.806000 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:15.805971 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1533e90b-5e64-4eda-b529-a11fc102b586" path="/var/lib/kubelet/pods/1533e90b-5e64-4eda-b529-a11fc102b586/volumes" Jul 07 00:26:16.046710 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:16.046674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" event={"ID":"a09f5c41-7ca1-4333-9339-433273efa70b","Type":"ContainerStarted","Data":"bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d"} Jul 07 00:26:16.046710 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:16.046712 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" event={"ID":"a09f5c41-7ca1-4333-9339-433273efa70b","Type":"ContainerStarted","Data":"e8f0b1ee5295a6a1dae4e45576814057af11e2e13da5419992573f4eb0e1f2b3"} Jul 07 00:26:16.063818 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:16.063716 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" podStartSLOduration=2.063701269 podStartE2EDuration="2.063701269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:26:16.061995246 +0000 UTC m=+2240.712417206" watchObservedRunningTime="2026-07-07 00:26:16.063701269 +0000 UTC m=+2240.714123216" Jul 07 00:26:25.015684 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:25.015646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:25.016039 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:25.015700 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:25.027803 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:25.027778 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:25.102943 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:25.102919 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:26:40.896000 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:40.895960 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6"] Jul 07 00:26:40.900179 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:40.900151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:40.902317 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:40.902285 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"kube-root-ca.crt\"" Jul 07 00:26:40.902455 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:40.902298 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"openshift-service-ca.crt\"" Jul 07 00:26:40.902792 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:40.902771 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"default-dockercfg-skmrn\"" Jul 07 00:26:40.902995 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:40.902797 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26a9dd\"/\"rate-test-model-26a9dd-kserve-self-signed-certs\"" Jul 07 00:26:40.906404 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:40.906386 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6"] Jul 07 00:26:41.043540 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.043479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103f057f-fd21-45a1-895d-fd17a604bafd-tls-certs\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.043540 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.043539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-dshm\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.043746 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.043602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-tmp-dir\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.043746 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.043631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-home\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.043746 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.043711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6r8c\" (UniqueName: \"kubernetes.io/projected/103f057f-fd21-45a1-895d-fd17a604bafd-kube-api-access-b6r8c\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.043861 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.043774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-model-cache\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.144866 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.144826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-tmp-dir\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145053 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.144873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-home\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145053 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.144925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6r8c\" (UniqueName: \"kubernetes.io/projected/103f057f-fd21-45a1-895d-fd17a604bafd-kube-api-access-b6r8c\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145053 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.144974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-model-cache\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145053 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.145045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103f057f-fd21-45a1-895d-fd17a604bafd-tls-certs\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145249 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.145069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-dshm\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145309 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.145261 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-tmp-dir\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145309 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.145267 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-home\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.145413 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.145324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-model-cache\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.147688 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.147628 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-dshm\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.147688 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.147666 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103f057f-fd21-45a1-895d-fd17a604bafd-tls-certs\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.152370 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.152349 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6r8c\" (UniqueName: \"kubernetes.io/projected/103f057f-fd21-45a1-895d-fd17a604bafd-kube-api-access-b6r8c\") pod \"rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.212414 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.212381 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:41.350830 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:41.350802 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6"] Jul 07 00:26:41.352911 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:26:41.352879 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod103f057f_fd21_45a1_895d_fd17a604bafd.slice/crio-dbb0da37291b51499b119065d0b40dcd48050ca7463794eb289be7482ae9b673 WatchSource:0}: Error finding container dbb0da37291b51499b119065d0b40dcd48050ca7463794eb289be7482ae9b673: Status 404 returned error can't find the container with id dbb0da37291b51499b119065d0b40dcd48050ca7463794eb289be7482ae9b673 Jul 07 00:26:42.180584 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:42.180552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" event={"ID":"103f057f-fd21-45a1-895d-fd17a604bafd","Type":"ContainerStarted","Data":"26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92"} Jul 07 00:26:42.180969 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:42.180593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" event={"ID":"103f057f-fd21-45a1-895d-fd17a604bafd","Type":"ContainerStarted","Data":"dbb0da37291b51499b119065d0b40dcd48050ca7463794eb289be7482ae9b673"} Jul 07 00:26:42.198091 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:42.198041 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" podStartSLOduration=2.198025713 podStartE2EDuration="2.198025713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:26:42.19606994 +0000 UTC m=+2266.846491901" watchObservedRunningTime="2026-07-07 00:26:42.198025713 +0000 UTC m=+2266.848447661" Jul 07 00:26:51.213391 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:51.213356 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:51.213912 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:51.213408 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:51.225714 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:51.225692 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:26:51.239660 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:26:51.239639 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:27:25.163884 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:25.163848 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9"] Jul 07 00:27:26.079894 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:26.079861 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6"] Jul 07 00:27:40.185166 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.185108 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" podUID="a09f5c41-7ca1-4333-9339-433273efa70b" containerName="main" containerID="cri-o://bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d" gracePeriod=45 Jul 07 00:27:40.438562 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.438507 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:27:40.484685 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.484656 2564 generic.go:356] "Generic (PLEG): container finished" podID="a09f5c41-7ca1-4333-9339-433273efa70b" containerID="bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d" exitCode=0 Jul 07 00:27:40.484853 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.484698 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" Jul 07 00:27:40.484853 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.484721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" event={"ID":"a09f5c41-7ca1-4333-9339-433273efa70b","Type":"ContainerDied","Data":"bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d"} Jul 07 00:27:40.484853 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.484753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9" event={"ID":"a09f5c41-7ca1-4333-9339-433273efa70b","Type":"ContainerDied","Data":"e8f0b1ee5295a6a1dae4e45576814057af11e2e13da5419992573f4eb0e1f2b3"} Jul 07 00:27:40.484853 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.484775 2564 scope.go:117] "RemoveContainer" containerID="bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d" Jul 07 00:27:40.493671 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.493646 2564 scope.go:117] "RemoveContainer" containerID="bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d" Jul 07 00:27:40.493926 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:27:40.493906 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d\": container with ID starting with bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d not found: ID does not exist" containerID="bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d" Jul 07 00:27:40.493969 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.493935 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d"} err="failed to get container status \"bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d\": rpc error: code = NotFound desc = could not find container \"bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d\": container with ID starting with bd46ce812d5bab1937919ca95d4f3f66ad64a2c95a9d9bfbfaae6e05a2bfd79d not found: ID does not exist" Jul 07 00:27:40.523836 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.523819 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-model-cache\") pod \"a09f5c41-7ca1-4333-9339-433273efa70b\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " Jul 07 00:27:40.523939 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.523844 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f5c41-7ca1-4333-9339-433273efa70b-tls-certs\") pod \"a09f5c41-7ca1-4333-9339-433273efa70b\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " Jul 07 00:27:40.523939 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.523899 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-dshm\") pod \"a09f5c41-7ca1-4333-9339-433273efa70b\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " Jul 07 00:27:40.524022 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.523940 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffdc8\" (UniqueName: \"kubernetes.io/projected/a09f5c41-7ca1-4333-9339-433273efa70b-kube-api-access-ffdc8\") pod \"a09f5c41-7ca1-4333-9339-433273efa70b\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " Jul 07 00:27:40.524022 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.523990 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-home\") pod \"a09f5c41-7ca1-4333-9339-433273efa70b\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " Jul 07 00:27:40.524022 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.524013 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-tmp-dir\") pod \"a09f5c41-7ca1-4333-9339-433273efa70b\" (UID: \"a09f5c41-7ca1-4333-9339-433273efa70b\") " Jul 07 00:27:40.524173 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.524096 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a09f5c41-7ca1-4333-9339-433273efa70b" (UID: "a09f5c41-7ca1-4333-9339-433273efa70b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:40.524310 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.524284 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a09f5c41-7ca1-4333-9339-433273efa70b" (UID: "a09f5c41-7ca1-4333-9339-433273efa70b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:40.524310 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.524296 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-home" (OuterVolumeSpecName: "home") pod "a09f5c41-7ca1-4333-9339-433273efa70b" (UID: "a09f5c41-7ca1-4333-9339-433273efa70b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:40.524432 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.524415 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:40.524478 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.524438 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:40.524478 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.524452 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:40.526070 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.526051 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-dshm" (OuterVolumeSpecName: "dshm") pod "a09f5c41-7ca1-4333-9339-433273efa70b" (UID: "a09f5c41-7ca1-4333-9339-433273efa70b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:40.526146 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.526118 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09f5c41-7ca1-4333-9339-433273efa70b-kube-api-access-ffdc8" (OuterVolumeSpecName: "kube-api-access-ffdc8") pod "a09f5c41-7ca1-4333-9339-433273efa70b" (UID: "a09f5c41-7ca1-4333-9339-433273efa70b"). InnerVolumeSpecName "kube-api-access-ffdc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:27:40.526146 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.526126 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09f5c41-7ca1-4333-9339-433273efa70b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a09f5c41-7ca1-4333-9339-433273efa70b" (UID: "a09f5c41-7ca1-4333-9339-433273efa70b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:27:40.625633 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.625612 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09f5c41-7ca1-4333-9339-433273efa70b-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:40.625633 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.625633 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09f5c41-7ca1-4333-9339-433273efa70b-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:40.625791 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.625642 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffdc8\" (UniqueName: \"kubernetes.io/projected/a09f5c41-7ca1-4333-9339-433273efa70b-kube-api-access-ffdc8\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:40.805416 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.805388 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9"] Jul 07 00:27:40.809145 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:40.809114 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a7bd37/rate-test-model-a7bd37-kserve-867bb84bf5-z86s9"] Jul 07 00:27:41.100766 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.100670 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" podUID="103f057f-fd21-45a1-895d-fd17a604bafd" containerName="main" containerID="cri-o://26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92" gracePeriod=45 Jul 07 00:27:41.230185 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.230138 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" podUID="103f057f-fd21-45a1-895d-fd17a604bafd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused" Jul 07 00:27:41.369065 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.369039 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:27:41.433516 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-dshm\") pod \"103f057f-fd21-45a1-895d-fd17a604bafd\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " Jul 07 00:27:41.433516 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-tmp-dir\") pod \"103f057f-fd21-45a1-895d-fd17a604bafd\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " Jul 07 00:27:41.433730 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433613 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6r8c\" (UniqueName: \"kubernetes.io/projected/103f057f-fd21-45a1-895d-fd17a604bafd-kube-api-access-b6r8c\") pod \"103f057f-fd21-45a1-895d-fd17a604bafd\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " Jul 07 00:27:41.433730 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103f057f-fd21-45a1-895d-fd17a604bafd-tls-certs\") pod \"103f057f-fd21-45a1-895d-fd17a604bafd\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " Jul 07 00:27:41.433730 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433713 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-home\") pod \"103f057f-fd21-45a1-895d-fd17a604bafd\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " Jul 07 00:27:41.433892 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433759 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-model-cache\") pod \"103f057f-fd21-45a1-895d-fd17a604bafd\" (UID: \"103f057f-fd21-45a1-895d-fd17a604bafd\") " Jul 07 00:27:41.433892 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433766 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "103f057f-fd21-45a1-895d-fd17a604bafd" (UID: "103f057f-fd21-45a1-895d-fd17a604bafd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:41.434019 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.433995 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-home" (OuterVolumeSpecName: "home") pod "103f057f-fd21-45a1-895d-fd17a604bafd" (UID: "103f057f-fd21-45a1-895d-fd17a604bafd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:41.434123 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.434102 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:41.434123 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.434122 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:41.434469 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.434117 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-model-cache" (OuterVolumeSpecName: "model-cache") pod "103f057f-fd21-45a1-895d-fd17a604bafd" (UID: "103f057f-fd21-45a1-895d-fd17a604bafd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:41.435911 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.435883 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103f057f-fd21-45a1-895d-fd17a604bafd-kube-api-access-b6r8c" (OuterVolumeSpecName: "kube-api-access-b6r8c") pod "103f057f-fd21-45a1-895d-fd17a604bafd" (UID: "103f057f-fd21-45a1-895d-fd17a604bafd"). InnerVolumeSpecName "kube-api-access-b6r8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:27:41.435992 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.435929 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103f057f-fd21-45a1-895d-fd17a604bafd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "103f057f-fd21-45a1-895d-fd17a604bafd" (UID: "103f057f-fd21-45a1-895d-fd17a604bafd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:27:41.436041 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.436024 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-dshm" (OuterVolumeSpecName: "dshm") pod "103f057f-fd21-45a1-895d-fd17a604bafd" (UID: "103f057f-fd21-45a1-895d-fd17a604bafd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:27:41.492294 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.492261 2564 generic.go:356] "Generic (PLEG): container finished" podID="103f057f-fd21-45a1-895d-fd17a604bafd" containerID="26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92" exitCode=0 Jul 07 00:27:41.492416 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.492305 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" Jul 07 00:27:41.492416 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.492347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" event={"ID":"103f057f-fd21-45a1-895d-fd17a604bafd","Type":"ContainerDied","Data":"26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92"} Jul 07 00:27:41.492416 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.492387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6" event={"ID":"103f057f-fd21-45a1-895d-fd17a604bafd","Type":"ContainerDied","Data":"dbb0da37291b51499b119065d0b40dcd48050ca7463794eb289be7482ae9b673"} Jul 07 00:27:41.492416 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.492403 2564 scope.go:117] "RemoveContainer" containerID="26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92" Jul 07 00:27:41.501413 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.501396 2564 scope.go:117] "RemoveContainer" containerID="26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92" Jul 07 00:27:41.501790 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:27:41.501769 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92\": container with ID starting with 26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92 not found: ID does not exist" containerID="26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92" Jul 07 00:27:41.501858 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.501798 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92"} err="failed to get container status \"26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92\": rpc error: code = NotFound desc = could not find container \"26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92\": container with ID starting with 26e54f76e369a4bf5721737dee4d60f81ed6bef7647ee0b0a2e6a5915bb6ba92 not found: ID does not exist" Jul 07 00:27:41.512716 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.512687 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6"] Jul 07 00:27:41.515103 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.515084 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-26a9dd/rate-test-model-26a9dd-kserve-56b7bf6cbc-gvfz6"] Jul 07 00:27:41.534706 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.534679 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103f057f-fd21-45a1-895d-fd17a604bafd-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:41.534706 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.534699 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:41.534706 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.534708 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103f057f-fd21-45a1-895d-fd17a604bafd-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:41.534921 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.534717 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6r8c\" (UniqueName: \"kubernetes.io/projected/103f057f-fd21-45a1-895d-fd17a604bafd-kube-api-access-b6r8c\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:27:41.806361 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.806325 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103f057f-fd21-45a1-895d-fd17a604bafd" path="/var/lib/kubelet/pods/103f057f-fd21-45a1-895d-fd17a604bafd/volumes" Jul 07 00:27:41.806738 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:27:41.806724 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09f5c41-7ca1-4333-9339-433273efa70b" path="/var/lib/kubelet/pods/a09f5c41-7ca1-4333-9339-433273efa70b/volumes" Jul 07 00:28:16.810806 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:16.810775 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g"] Jul 07 00:28:16.811338 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:16.810993 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" containerID="cri-o://504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01" gracePeriod=30 Jul 07 00:28:22.071662 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.071583 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:28:22.079648 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.079629 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lvpzw_a35e657e-8930-4d1a-952c-1ecc10f311a5/manager/0.log" Jul 07 00:28:22.197764 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.197738 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mx9r9_d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c/manager/0.log" Jul 07 00:28:22.218600 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218571 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8zbj\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-kube-api-access-c8zbj\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218752 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218638 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-data\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218752 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218684 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istiod-ca-cert\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218752 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218715 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-podinfo\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218752 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218741 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-certs\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218974 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218772 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-socket\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218974 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-credential-socket\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218974 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-envoy\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.218974 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218913 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-token\") pod \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\" (UID: \"7c8cfa35-10ec-4b88-858d-37fa4ecac12e\") " Jul 07 00:28:22.219175 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.218992 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:22.219175 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.219043 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:22.219175 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.219056 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:22.219350 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.219248 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:28:22.219350 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.219308 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.219350 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.219325 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.219350 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.219339 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.219887 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.219582 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:22.221138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.221106 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:28:22.221138 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.221119 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:22.221488 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.221466 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-kube-api-access-c8zbj" (OuterVolumeSpecName: "kube-api-access-c8zbj") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "kube-api-access-c8zbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:28:22.221488 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.221486 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c8cfa35-10ec-4b88-858d-37fa4ecac12e" (UID: "7c8cfa35-10ec-4b88-858d-37fa4ecac12e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:28:22.320623 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.320589 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8zbj\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-kube-api-access-c8zbj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.320623 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.320617 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.320623 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.320627 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.320907 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.320635 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.320907 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.320642 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.320907 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.320650 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8cfa35-10ec-4b88-858d-37fa4ecac12e-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 07 00:28:22.706049 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.706011 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerID="504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01" exitCode=0 Jul 07 00:28:22.706246 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.706057 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" Jul 07 00:28:22.706246 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.706083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" event={"ID":"7c8cfa35-10ec-4b88-858d-37fa4ecac12e","Type":"ContainerDied","Data":"504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01"} Jul 07 00:28:22.706246 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.706136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g" event={"ID":"7c8cfa35-10ec-4b88-858d-37fa4ecac12e","Type":"ContainerDied","Data":"bf67f26456c4966d1dd5b331197453e65717bc3c7e695a564c92e3a6343ec4ae"} Jul 07 00:28:22.706246 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.706163 2564 scope.go:117] "RemoveContainer" containerID="504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01" Jul 07 00:28:22.715349 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.715333 2564 scope.go:117] "RemoveContainer" containerID="d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769" Jul 07 00:28:22.723205 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.723188 2564 scope.go:117] "RemoveContainer" containerID="504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01" Jul 07 00:28:22.723435 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:28:22.723417 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01\": container with ID starting with 504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01 not found: ID does not exist" containerID="504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01" Jul 07 00:28:22.723476 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.723444 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01"} err="failed to get container status \"504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01\": rpc error: code = NotFound desc = could not find container \"504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01\": container with ID starting with 504d3c383b81d84aebd008be7cb8837d96e018980f6609aab3069fbab99b3e01 not found: ID does not exist" Jul 07 00:28:22.723476 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.723460 2564 scope.go:117] "RemoveContainer" containerID="d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769" Jul 07 00:28:22.723709 ip-10-0-139-100 kubenswrapper[2564]: E0707 00:28:22.723691 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769\": container with ID starting with d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769 not found: ID does not exist" containerID="d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769" Jul 07 00:28:22.723750 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.723716 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769"} err="failed to get container status \"d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769\": rpc error: code = NotFound desc = could not find container \"d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769\": container with ID starting with d06995ea8c73371148056b918880e8229a152a5ed95ccecddc03d2bd91533769 not found: ID does not exist" Jul 07 00:28:22.728230 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.728207 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g"] Jul 07 00:28:22.735700 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:22.735676 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-26a9dd-openshift-default-56695db899-nc75g"] Jul 07 00:28:23.238180 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:23.238150 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64648fd55c-z7rpx_bf10b018-5df1-4bda-82bd-64f925fbcf6e/copy-manifests/0.log" Jul 07 00:28:23.256716 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:23.256675 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64648fd55c-z7rpx_bf10b018-5df1-4bda-82bd-64f925fbcf6e/manager/0.log" Jul 07 00:28:23.805162 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:23.805132 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" path="/var/lib/kubelet/pods/7c8cfa35-10ec-4b88-858d-37fa4ecac12e/volumes" Jul 07 00:28:24.759555 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:24.759520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-g2g5m_1fc5f808-c2f8-4daa-9783-118e3e2fd530/manager/0.log" Jul 07 00:28:24.874249 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:24.874218 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-sh4d6_618e1dd4-2671-45c4-9f98-f67aefc3edda/manager/0.log" Jul 07 00:28:25.104303 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:25.104235 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-9r6v4_51eaaf9d-3481-4948-a048-5db2804d25f8/registry-server/0.log" Jul 07 00:28:25.235634 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:25.235597 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bpsbr_78cd98e8-58ab-43c3-9969-0aa78b3df6ed/manager/0.log" Jul 07 00:28:25.818771 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:25.818739 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4prhvh_fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8/istio-proxy/0.log" Jul 07 00:28:26.618661 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:26.618627 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d79d85-rmpz9_beb31d50-ff75-48a0-afc9-42ce85ddf86d/router/0.log" Jul 07 00:28:27.071335 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:27.071310 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-b4xtq_d5a60c67-6895-49d9-9b8a-26bc3ba91804/main/0.log" Jul 07 00:28:27.300622 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:27.300595 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhmg6m_23e5d46c-45b8-4c5c-ab59-b6a77ea96fce/main/0.log" Jul 07 00:28:27.409945 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:27.409918 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6x46h_b6e1bc54-e7d7-40f8-8cfc-165e649d5f0c/main/0.log" Jul 07 00:28:34.457651 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:34.457618 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zlnnn_1a7f4dcc-d175-4d88-a24f-6f9c3cf2b606/global-pull-secret-syncer/0.log" Jul 07 00:28:34.576368 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:34.576331 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s5wl5_5352efee-cf3f-463a-82ff-8e9cb779e171/konnectivity-agent/0.log" Jul 07 00:28:34.647563 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:34.647538 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-100.ec2.internal_319f82ace8919ccdff136f05eb05d345/haproxy/0.log" Jul 07 00:28:40.016589 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:40.016557 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-g2g5m_1fc5f808-c2f8-4daa-9783-118e3e2fd530/manager/0.log" Jul 07 00:28:40.038509 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:40.038474 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-sh4d6_618e1dd4-2671-45c4-9f98-f67aefc3edda/manager/0.log" Jul 07 00:28:40.109506 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:40.109455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-9r6v4_51eaaf9d-3481-4948-a048-5db2804d25f8/registry-server/0.log" Jul 07 00:28:40.304259 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:40.304178 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bpsbr_78cd98e8-58ab-43c3-9969-0aa78b3df6ed/manager/0.log" Jul 07 00:28:42.271477 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.271439 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-tqtxf_d7cca9a6-fb2f-4129-ab5a-9846d7ca9b16/cluster-monitoring-operator/0.log" Jul 07 00:28:42.387425 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.387392 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bd45d5848-ssflm_7fcf0797-a1e3-4ae2-949f-3a96fcf52453/metrics-server/0.log" Jul 07 00:28:42.416561 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.416538 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-pl2c7_dbf032d3-63ac-41e3-b3ea-1145e00556b1/monitoring-plugin/0.log" Jul 07 00:28:42.627343 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.627314 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbc9q_75cceadc-045e-461f-b931-ba21d5855cc0/node-exporter/0.log" Jul 07 00:28:42.643832 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.643764 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbc9q_75cceadc-045e-461f-b931-ba21d5855cc0/kube-rbac-proxy/0.log" Jul 07 00:28:42.661632 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.661611 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbc9q_75cceadc-045e-461f-b931-ba21d5855cc0/init-textfile/0.log" Jul 07 00:28:42.772038 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.772010 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2aa98467-de04-4a8f-87d8-989121144d81/prometheus/0.log" Jul 07 00:28:42.791832 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.791805 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2aa98467-de04-4a8f-87d8-989121144d81/config-reloader/0.log" Jul 07 00:28:42.813815 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.813796 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2aa98467-de04-4a8f-87d8-989121144d81/thanos-sidecar/0.log" Jul 07 00:28:42.833222 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.833187 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2aa98467-de04-4a8f-87d8-989121144d81/kube-rbac-proxy-web/0.log" Jul 07 00:28:42.853748 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.853724 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2aa98467-de04-4a8f-87d8-989121144d81/kube-rbac-proxy/0.log" Jul 07 00:28:42.879122 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.879090 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2aa98467-de04-4a8f-87d8-989121144d81/kube-rbac-proxy-thanos/0.log" Jul 07 00:28:42.912305 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:42.912226 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2aa98467-de04-4a8f-87d8-989121144d81/init-config-reloader/0.log" Jul 07 00:28:43.087942 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:43.087897 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68854b46-x79cd_6442b11f-8354-4648-9b52-450a281f6033/thanos-query/0.log" Jul 07 00:28:43.113523 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:43.113469 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68854b46-x79cd_6442b11f-8354-4648-9b52-450a281f6033/kube-rbac-proxy-web/0.log" Jul 07 00:28:43.133756 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:43.133735 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68854b46-x79cd_6442b11f-8354-4648-9b52-450a281f6033/kube-rbac-proxy/0.log" Jul 07 00:28:43.154919 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:43.154892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68854b46-x79cd_6442b11f-8354-4648-9b52-450a281f6033/prom-label-proxy/0.log" Jul 07 00:28:43.178895 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:43.178806 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68854b46-x79cd_6442b11f-8354-4648-9b52-450a281f6033/kube-rbac-proxy-rules/0.log" Jul 07 00:28:43.202604 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:43.202583 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c68854b46-x79cd_6442b11f-8354-4648-9b52-450a281f6033/kube-rbac-proxy-metrics/0.log" Jul 07 00:28:44.790674 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.790643 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844"] Jul 07 00:28:44.791048 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791026 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09f5c41-7ca1-4333-9339-433273efa70b" containerName="main" Jul 07 00:28:44.791048 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791038 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09f5c41-7ca1-4333-9339-433273efa70b" containerName="main" Jul 07 00:28:44.791048 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791048 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103f057f-fd21-45a1-895d-fd17a604bafd" containerName="main" Jul 07 00:28:44.791151 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791054 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="103f057f-fd21-45a1-895d-fd17a604bafd" containerName="main" Jul 07 00:28:44.791151 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791070 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" Jul 07 00:28:44.791151 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791076 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" Jul 07 00:28:44.791304 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791161 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" Jul 07 00:28:44.791304 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791168 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="103f057f-fd21-45a1-895d-fd17a604bafd" containerName="main" Jul 07 00:28:44.791304 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791173 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09f5c41-7ca1-4333-9339-433273efa70b" containerName="main" Jul 07 00:28:44.791304 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.791180 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8cfa35-10ec-4b88-858d-37fa4ecac12e" containerName="istio-proxy" Jul 07 00:28:44.794511 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.794476 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.796818 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.796794 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f7d67\"/\"kube-root-ca.crt\"" Jul 07 00:28:44.796932 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.796819 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f7d67\"/\"default-dockercfg-bpjqr\"" Jul 07 00:28:44.796932 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.796794 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f7d67\"/\"openshift-service-ca.crt\"" Jul 07 00:28:44.801915 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.801898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844"] Jul 07 00:28:44.829555 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.829527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-lib-modules\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.829685 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.829624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-proc\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.829685 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.829645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-sys\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.829685 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.829660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-podres\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.829795 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.829733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhdrq\" (UniqueName: \"kubernetes.io/projected/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-kube-api-access-lhdrq\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931012 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.930979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-proc\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931012 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-sys\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-podres\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhdrq\" (UniqueName: \"kubernetes.io/projected/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-kube-api-access-lhdrq\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931099 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-proc\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931105 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-sys\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931189 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-lib-modules\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931345 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931211 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-podres\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.931345 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.931311 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-lib-modules\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:44.939154 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:44.939121 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhdrq\" (UniqueName: \"kubernetes.io/projected/6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941-kube-api-access-lhdrq\") pod \"perf-node-gather-daemonset-c8844\" (UID: \"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941\") " pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:45.078913 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.078814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:28:45.088201 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.088169 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/2.log" Jul 07 00:28:45.105935 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.105914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:45.239653 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.239631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844"] Jul 07 00:28:45.241848 ip-10-0-139-100 kubenswrapper[2564]: W0707 00:28:45.241821 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a09aca2_3ff5_4c2c_8bb4_945d8fdbb941.slice/crio-038c17092ddc77ac059a7fee64bedf0f28d1405d1484cb38037569f4c5dcfbfc WatchSource:0}: Error finding container 038c17092ddc77ac059a7fee64bedf0f28d1405d1484cb38037569f4c5dcfbfc: Status 404 returned error can't find the container with id 038c17092ddc77ac059a7fee64bedf0f28d1405d1484cb38037569f4c5dcfbfc Jul 07 00:28:45.688348 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.688324 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-555fdc4c8c-bwjvn_c7223f9b-03b6-46dd-9396-8fab3bbceffd/console/0.log" Jul 07 00:28:45.826009 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.825971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" event={"ID":"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941","Type":"ContainerStarted","Data":"559c64921bb0431efcc948c6b8cac2d51521e9a54207675fbae58ae3b486aa06"} Jul 07 00:28:45.826380 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.826031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" event={"ID":"6a09aca2-3ff5-4c2c-8bb4-945d8fdbb941","Type":"ContainerStarted","Data":"038c17092ddc77ac059a7fee64bedf0f28d1405d1484cb38037569f4c5dcfbfc"} Jul 07 00:28:45.826380 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.826060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:45.848649 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:45.848604 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" podStartSLOduration=1.8485884559999999 podStartE2EDuration="1.848588456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 00:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:28:45.845683221 +0000 UTC m=+2390.496105192" watchObservedRunningTime="2026-07-07 00:28:45.848588456 +0000 UTC m=+2390.499010454" Jul 07 00:28:46.389831 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:46.389790 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-c9nx8_49c1d9cf-b5e4-47b5-ae63-77ea95565764/volume-data-source-validator/0.log" Jul 07 00:28:47.515286 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:47.515251 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jn4x7_9cf530db-a95a-469f-9bde-41d882c205ab/dns/0.log" Jul 07 00:28:47.549154 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:47.549127 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jn4x7_9cf530db-a95a-469f-9bde-41d882c205ab/kube-rbac-proxy/0.log" Jul 07 00:28:47.737611 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:47.737587 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5kssz_f9e3d2fc-85d2-4071-a95b-9ca5be60a285/dns-node-resolver/0.log" Jul 07 00:28:48.527051 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:48.527021 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9ds8_1ef88259-880c-4c27-a7c5-500f8b1f2698/node-ca/0.log" Jul 07 00:28:49.486328 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:49.486287 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4prhvh_fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8/istio-proxy/0.log" Jul 07 00:28:49.802766 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:49.802649 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d79d85-rmpz9_beb31d50-ff75-48a0-afc9-42ce85ddf86d/router/0.log" Jul 07 00:28:50.507075 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:50.507047 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7kbvg_4529a4c7-0153-476f-bf94-566619bd5d22/serve-healthcheck-canary/0.log" Jul 07 00:28:51.068951 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:51.068921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xgj9q_bdd76a4e-600e-458e-adb3-801d5f30c4be/insights-operator/0.log" Jul 07 00:28:51.069672 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:51.069653 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xgj9q_bdd76a4e-600e-458e-adb3-801d5f30c4be/insights-operator/1.log" Jul 07 00:28:51.092281 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:51.092258 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wsvn_75cc4fb0-79f4-45ab-98ed-bf1038399bc2/kube-rbac-proxy/0.log" Jul 07 00:28:51.111585 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:51.111562 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wsvn_75cc4fb0-79f4-45ab-98ed-bf1038399bc2/exporter/0.log" Jul 07 00:28:51.132169 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:51.132148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wsvn_75cc4fb0-79f4-45ab-98ed-bf1038399bc2/extractor/0.log" Jul 07 00:28:51.841200 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:51.841169 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f7d67/perf-node-gather-daemonset-c8844" Jul 07 00:28:53.520922 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:53.520892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lvpzw_a35e657e-8930-4d1a-952c-1ecc10f311a5/manager/0.log" Jul 07 00:28:53.757777 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:53.757697 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mx9r9_d3eb8e3c-a5cd-46ba-a641-9a339fc2e00c/manager/0.log" Jul 07 00:28:54.301069 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:54.301034 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64648fd55c-z7rpx_bf10b018-5df1-4bda-82bd-64f925fbcf6e/manager/0.log" Jul 07 00:28:54.318799 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:54.318771 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64648fd55c-z7rpx_bf10b018-5df1-4bda-82bd-64f925fbcf6e/copy-manifests/0.log" Jul 07 00:28:55.555092 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:55.555060 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4prhvh_fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8/istio-proxy/0.log" Jul 07 00:28:55.705267 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:55.705241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f447c9d54-k7sjx_56fc5e67-0301-4ad4-adeb-dd1beb086775/manager/0.log" Jul 07 00:28:55.810230 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:55.810153 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d79d85-rmpz9_beb31d50-ff75-48a0-afc9-42ce85ddf86d/router/0.log" Jul 07 00:28:55.922048 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:55.922025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:28:55.933360 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:55.933332 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-lz94n_a3acadf3-310e-4f27-99ed-c62ee39567a1/console-operator/1.log" Jul 07 00:28:57.019453 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:57.019425 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4prhvh_fb5d3f56-d9a0-4d0b-a112-6cd21e83c5c8/istio-proxy/0.log" Jul 07 00:28:57.292667 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:28:57.292593 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d79d85-rmpz9_beb31d50-ff75-48a0-afc9-42ce85ddf86d/router/0.log" Jul 07 00:29:00.974735 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:00.974644 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-hbn4z_8e28fdd5-ea71-4ec9-a304-4dff04bf11f3/kube-storage-version-migrator-operator/1.log" Jul 07 00:29:00.976447 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:00.976414 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-hbn4z_8e28fdd5-ea71-4ec9-a304-4dff04bf11f3/kube-storage-version-migrator-operator/0.log" Jul 07 00:29:02.139849 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:02.139819 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttrjd_13605fab-a33b-41d6-8f26-0c2defe642af/kube-multus-additional-cni-plugins/0.log" Jul 07 00:29:02.160097 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:02.160064 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttrjd_13605fab-a33b-41d6-8f26-0c2defe642af/egress-router-binary-copy/0.log" Jul 07 00:29:02.180148 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:02.180119 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttrjd_13605fab-a33b-41d6-8f26-0c2defe642af/cni-plugins/0.log" Jul 07 00:29:02.196962 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:02.196935 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttrjd_13605fab-a33b-41d6-8f26-0c2defe642af/bond-cni-plugin/0.log" Jul 07 00:29:02.214984 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:02.214954 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttrjd_13605fab-a33b-41d6-8f26-0c2defe642af/routeoverride-cni/0.log" Jul 07 00:29:02.233166 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:02.233142 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttrjd_13605fab-a33b-41d6-8f26-0c2defe642af/whereabouts-cni-bincopy/0.log" Jul 07 00:29:02.252332 ip-10-0-139-100 kubenswrapper[2564]: I0707 00:29:02.252316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ttrjd_13605fab-a33b-41d6-8f26-0c2defe642af/whereabouts-cni/0.log"