Jun 10 10:44:17.217108 ip-10-0-138-231 systemd[1]: Starting Kubernetes Kubelet... Jun 10 10:44:17.679045 ip-10-0-138-231 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 10 10:44:17.679045 ip-10-0-138-231 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jun 10 10:44:17.679045 ip-10-0-138-231 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 10 10:44:17.679045 ip-10-0-138-231 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jun 10 10:44:17.679045 ip-10-0-138-231 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 10 10:44:17.679602 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.679106 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jun 10 10:44:17.684084 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684068 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:17.684084 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684084 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684089 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684092 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684095 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684098 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684100 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684103 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684106 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684109 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684111 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684114 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684116 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684119 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684122 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684124 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684130 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684132 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684135 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684137 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684140 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:17.684146 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684142 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684145 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684147 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684150 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684152 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684155 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684158 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684160 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684163 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684165 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684167 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684170 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684172 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684175 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684177 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684180 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684182 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684185 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684187 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684190 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:17.684618 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684192 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684195 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684198 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684200 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684203 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684205 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684210 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684214 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684218 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684221 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684224 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684226 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684229 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684232 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684235 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684238 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684241 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684244 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:17.685120 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684247 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684250 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684253 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684256 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684259 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684262 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684265 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684269 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684272 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684275 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684277 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684280 2578 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684282 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684285 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684287 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684289 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684292 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684297 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684299 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:17.685557 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684301 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684304 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684306 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684309 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684311 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684314 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684316 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684318 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684321 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684325 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684327 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684330 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684332 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684335 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684337 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684340 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684342 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684345 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684347 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684349 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:17.686021 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.684352 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685745 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685754 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685758 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685761 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685764 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685767 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685770 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685772 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685775 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685778 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685781 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685783 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685786 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685788 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685791 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685808 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685811 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685813 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685815 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:17.686495 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685818 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685821 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685824 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685826 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685829 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685832 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685834 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685837 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685839 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685842 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685845 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685847 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685849 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685852 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685854 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685857 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685859 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685862 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685864 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685867 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:17.687043 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685870 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685872 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685874 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685877 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685880 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685883 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685885 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685887 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685890 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685892 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685895 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685897 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685900 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685903 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685906 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685909 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685911 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685914 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685916 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685919 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:17.687529 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685921 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685924 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685926 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685929 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685932 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685935 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685937 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685940 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685942 2578 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685945 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685948 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685952 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685956 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685959 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685962 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685964 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685968 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685970 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685973 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:17.688062 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685976 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685979 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685981 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685984 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685987 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685989 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685993 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685995 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.685998 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686000 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686003 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686005 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686008 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686011 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686013 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686016 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686018 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686021 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686023 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:17.688515 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686026 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686029 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686416 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686423 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686426 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686429 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686432 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686435 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686438 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686440 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686443 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686446 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686449 2578 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686451 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686453 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686456 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686458 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686461 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686463 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:17.688997 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686467 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686470 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686472 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686475 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686477 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686480 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686482 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686485 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686487 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686489 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686492 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686495 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686497 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686500 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686503 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686505 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686508 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686510 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686513 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686515 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:17.689485 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686518 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686520 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686523 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686525 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686528 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686531 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686534 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686537 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686540 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686542 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686545 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686547 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686555 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686558 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686561 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686563 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686566 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686568 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686571 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686573 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686576 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:17.689977 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686579 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686581 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686584 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686586 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686589 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686592 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686607 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686610 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686614 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686618 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686622 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686624 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686627 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686630 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686634 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686638 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686641 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686644 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:17.690513 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686646 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686649 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686651 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686654 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686657 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686661 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686663 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686666 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686669 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686671 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686674 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686676 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686679 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686681 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686684 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686686 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686689 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686691 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686694 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686696 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:17.690975 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686699 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686701 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.686704 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686788 2578 flags.go:64] FLAG: --address="0.0.0.0" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686812 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686818 2578 flags.go:64] FLAG: --anonymous-auth="true" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686822 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686827 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686831 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686838 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686844 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686850 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686855 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686858 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686862 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686865 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686868 2578 flags.go:64] FLAG: --cgroup-root="" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686874 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686877 2578 flags.go:64] FLAG: --client-ca-file="" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686880 2578 flags.go:64] FLAG: --cloud-provider="external" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686883 2578 flags.go:64] FLAG: --cluster-dns="[]" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686888 2578 flags.go:64] FLAG: --cluster-domain="" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686891 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686894 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jun 10 10:44:17.691456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686896 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686900 2578 flags.go:64] FLAG: --container-log-max-files="5" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686903 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686906 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686909 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686912 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686915 2578 flags.go:64] FLAG: --contention-profiling="false" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686918 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686921 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686924 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686927 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686931 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686934 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686936 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686939 2578 flags.go:64] FLAG: --enable-load-reader="false" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686942 2578 flags.go:64] FLAG: --enable-server="true" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686944 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686948 2578 flags.go:64] FLAG: --event-burst="100" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686952 2578 flags.go:64] FLAG: --event-qps="50" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686954 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686958 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686960 2578 flags.go:64] FLAG: --eviction-hard="" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686964 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686967 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686970 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jun 10 10:44:17.692099 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686975 2578 flags.go:64] FLAG: --eviction-soft="" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686978 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686980 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686983 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686986 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686989 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686992 2578 flags.go:64] FLAG: --fail-swap-on="true" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686995 2578 flags.go:64] FLAG: --feature-gates="" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.686999 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687002 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687005 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687008 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687011 2578 flags.go:64] FLAG: --healthz-port="10248" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687014 2578 flags.go:64] FLAG: --help="false" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687017 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687020 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687023 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687025 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687029 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687032 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687035 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687037 2578 flags.go:64] FLAG: --image-service-endpoint="" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687040 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jun 10 10:44:17.692681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687042 2578 flags.go:64] FLAG: --kube-api-burst="100" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687045 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687048 2578 flags.go:64] FLAG: --kube-api-qps="50" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687051 2578 flags.go:64] FLAG: --kube-reserved="" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687054 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687057 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687060 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687063 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687066 2578 flags.go:64] FLAG: --lock-file="" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687071 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687074 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687077 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687082 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687085 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687088 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687090 2578 flags.go:64] FLAG: --logging-format="text" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687093 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687096 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687099 2578 flags.go:64] FLAG: --manifest-url="" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687101 2578 flags.go:64] FLAG: --manifest-url-header="" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687106 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687108 2578 flags.go:64] FLAG: --max-open-files="1000000" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687112 2578 flags.go:64] FLAG: --max-pods="110" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687115 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687118 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jun 10 10:44:17.693286 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687121 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687123 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687126 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687129 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687132 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687140 2578 flags.go:64] FLAG: --node-status-max-images="50" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687142 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687145 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687148 2578 flags.go:64] FLAG: --pod-cidr="" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687151 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746976bfb3a92b93dacc79c151b98fd4c932eef19c3e2c1516b14eebcac9184d" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687156 2578 flags.go:64] FLAG: --pod-manifest-path="" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687159 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687162 2578 flags.go:64] FLAG: --pods-per-core="0" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687166 2578 flags.go:64] FLAG: --port="10250" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687169 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687171 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060fbe946061e32ac" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687175 2578 flags.go:64] FLAG: --qos-reserved="" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687180 2578 flags.go:64] FLAG: --read-only-port="10255" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687183 2578 flags.go:64] FLAG: --register-node="true" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687186 2578 flags.go:64] FLAG: --register-with-taints="" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687189 2578 flags.go:64] FLAG: --registry-burst="10" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687192 2578 flags.go:64] FLAG: --registry-qps="5" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687194 2578 flags.go:64] FLAG: --reserved-cpus="" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687197 2578 flags.go:64] FLAG: --reserved-memory="" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687201 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jun 10 10:44:17.694058 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687204 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687207 2578 flags.go:64] FLAG: --rotate-certificates="false" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687210 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687212 2578 flags.go:64] FLAG: --runonce="false" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687215 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687218 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687221 2578 flags.go:64] FLAG: --seccomp-default="false" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687223 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687226 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687229 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687232 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687235 2578 flags.go:64] FLAG: --storage-driver-password="root" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687238 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687240 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687243 2578 flags.go:64] FLAG: --storage-driver-user="root" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687246 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687249 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687251 2578 flags.go:64] FLAG: --system-cgroups="" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687254 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687259 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687262 2578 flags.go:64] FLAG: --tls-cert-file="" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687269 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687273 2578 flags.go:64] FLAG: --tls-min-version="" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687277 2578 flags.go:64] FLAG: --tls-private-key-file="" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687282 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jun 10 10:44:17.694949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687284 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687287 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687291 2578 flags.go:64] FLAG: --v="2" Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687295 2578 flags.go:64] FLAG: --version="false" Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687298 2578 flags.go:64] FLAG: --vmodule="" Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687302 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.687306 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687437 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687441 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687444 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687447 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687450 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687453 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687456 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687458 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687461 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687463 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687466 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687468 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687471 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687473 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:17.695566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687476 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687478 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687481 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687483 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687486 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687488 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687490 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687493 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687496 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687500 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687504 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687506 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687509 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687511 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687514 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687516 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687519 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687521 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687524 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687526 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:17.696138 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687528 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687531 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687533 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687536 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687539 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687541 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687544 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687555 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687558 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687560 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687563 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687565 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687568 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687570 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687573 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687575 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687578 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687581 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687583 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687585 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:17.696639 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687588 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687592 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687595 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687598 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687600 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687603 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687605 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687609 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687613 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687616 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687619 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687621 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687624 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687628 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687631 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687634 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687636 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687639 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687641 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:17.697188 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687644 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687646 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687648 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687651 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687653 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687656 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687658 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687662 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687666 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687668 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687671 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687674 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687677 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687679 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687682 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687685 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687687 2578 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687690 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687692 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:17.697655 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687695 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687697 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687700 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687702 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687705 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687707 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.687709 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.688424 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.696171 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.696187 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.696205 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.696212 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696270 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696275 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696278 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:17.698145 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696281 2578 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696283 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696287 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696290 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696293 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696296 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696298 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696301 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696304 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696307 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696309 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696312 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696315 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696317 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696320 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696323 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696327 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696331 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696346 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:17.698523 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696349 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696352 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696355 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696357 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696360 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696362 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696365 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696368 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696370 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696373 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696375 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696378 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696380 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696383 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696385 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696388 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696391 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696393 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696396 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696398 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:17.699005 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696401 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696403 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696406 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696408 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696411 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696414 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696416 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696419 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696422 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696425 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696427 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696430 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696433 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696436 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696438 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696441 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696444 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696448 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696451 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696453 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:17.699589 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696456 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696459 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696462 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696464 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696467 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696470 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696472 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696475 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696477 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696480 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696483 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696485 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696487 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696490 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696492 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696495 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696497 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696501 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696504 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:17.700097 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696508 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696511 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696514 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696516 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696519 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696522 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696524 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696527 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696529 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696532 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696535 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696537 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696540 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696542 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696544 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696547 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696549 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:17.700562 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696552 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.696557 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696675 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696680 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696683 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696686 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696689 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696691 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696695 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696697 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696700 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696702 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696705 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696707 2578 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696711 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:17.701169 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696714 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696717 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696719 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696723 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696725 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696728 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696731 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696733 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696736 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696738 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696741 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696744 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696747 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696750 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696754 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696756 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696759 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696763 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696766 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:17.701552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696769 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696772 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696775 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696777 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696780 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696783 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696785 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696788 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696790 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696792 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696814 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696817 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696820 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696824 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696826 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696829 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696832 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696834 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696837 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696839 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:17.702066 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696842 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696844 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696847 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696849 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696852 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696854 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696857 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696860 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696862 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696864 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696867 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696871 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696874 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696877 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696880 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696882 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696885 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696888 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696890 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:17.702553 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696893 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696895 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696898 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696900 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696903 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696905 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696907 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696911 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696914 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696916 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696919 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696922 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696924 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696927 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696929 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696932 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696934 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696936 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696939 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696942 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:17.703011 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696944 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696947 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696949 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696951 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696954 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696956 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696959 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:17.696961 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.696966 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.697077 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.700012 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.701531 2578 server.go:1019] "Starting client certificate rotation" Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.701636 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 10 10:44:17.703493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.701665 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 10 10:44:17.704704 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.704689 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 10 10:44:17.706176 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.706160 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 10 10:44:17.717762 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.717745 2578 log.go:25] "Validated CRI v1 runtime API" Jun 10 10:44:17.722144 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.722127 2578 log.go:25] "Validated CRI v1 image API" Jun 10 10:44:17.722917 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.722904 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jun 10 10:44:17.727108 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.727086 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e46faf8-5eec-41ad-8eb6-460ec6db84ce:/dev/nvme0n1p4 d41119b7-62bf-493c-af46-a9b39e2d0469:/dev/nvme0n1p3] Jun 10 10:44:17.727158 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.727109 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jun 10 10:44:17.734524 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.734408 2578 manager.go:216] Machine: {Timestamp:2026-06-10 10:44:17.73306676 +0000 UTC m=+0.382122743 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101774 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec207ed6b8742ac7e6b8eeb2a7e489a0 SystemUUID:ec207ed6-b874-2ac7-e6b8-eeb2a7e489a0 BootID:76796e39-abba-4848-8c51-4ca2d56e9c2f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:40:8c:99:37:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:40:8c:99:37:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:c6:cb:8d:45:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jun 10 10:44:17.734524 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.734509 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jun 10 10:44:17.734689 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.734597 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.118.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260602-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jun 10 10:44:17.735642 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.735623 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 10 10:44:17.738176 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.738152 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jun 10 10:44:17.738310 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.738180 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-231.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jun 10 10:44:17.738994 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.738983 2578 topology_manager.go:138] "Creating topology manager with none policy" Jun 10 10:44:17.739031 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.738997 2578 container_manager_linux.go:309] "Creating device plugin manager" Jun 10 10:44:17.739031 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.739008 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 10 10:44:17.739755 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.739745 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 10 10:44:17.740680 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.740670 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jun 10 10:44:17.744027 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744015 2578 state_mem.go:36] "Initialized new in-memory state store" Jun 10 10:44:17.744152 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744142 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jun 10 10:44:17.744190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744176 2578 kubelet.go:486] "Attempting to sync node with API server" Jun 10 10:44:17.744190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744188 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jun 10 10:44:17.744243 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744204 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jun 10 10:44:17.744243 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744213 2578 kubelet.go:398] "Adding apiserver pod source" Jun 10 10:44:17.744243 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744223 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jun 10 10:44:17.744589 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744577 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 10 10:44:17.744654 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.744595 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 10 10:44:17.747204 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.747183 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jun 10 10:44:17.748895 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.748881 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jun 10 10:44:17.748965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.748912 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jun 10 10:44:17.750711 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750700 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jun 10 10:44:17.750754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750717 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jun 10 10:44:17.750754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750724 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jun 10 10:44:17.750754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750730 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jun 10 10:44:17.750754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750739 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jun 10 10:44:17.750754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750745 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jun 10 10:44:17.750754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750752 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jun 10 10:44:17.750754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750757 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jun 10 10:44:17.750958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750765 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jun 10 10:44:17.750958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750771 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jun 10 10:44:17.750958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750779 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jun 10 10:44:17.750958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.750784 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jun 10 10:44:17.751660 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.751650 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jun 10 10:44:17.751660 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.751660 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jun 10 10:44:17.754966 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.754946 2578 server.go:1268] "Started kubelet" Jun 10 10:44:17.755122 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.755032 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jun 10 10:44:17.755180 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.755106 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jun 10 10:44:17.755225 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.755179 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jun 10 10:44:17.755915 ip-10-0-138-231 systemd[1]: Started Kubernetes Kubelet. Jun 10 10:44:17.757887 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.757871 2578 server.go:310] "Adding debug handlers to kubelet server" Jun 10 10:44:17.758584 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.758538 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jun 10 10:44:17.758702 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.758586 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-231.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jun 10 10:44:17.759306 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.759276 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jun 10 10:44:17.759376 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.759363 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jun 10 10:44:17.763929 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.763895 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jun 10 10:44:17.764762 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.764749 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jun 10 10:44:17.764845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.764767 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jun 10 10:44:17.764907 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.764860 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:17.765478 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765456 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jun 10 10:44:17.765578 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765506 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jun 10 10:44:17.765578 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765518 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jun 10 10:44:17.765668 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765587 2578 factory.go:55] Registering systemd factory Jun 10 10:44:17.765668 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765602 2578 factory.go:223] Registration of the systemd container factory successfully Jun 10 10:44:17.765668 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765665 2578 reconstruct.go:97] "Volume reconstruction finished" Jun 10 10:44:17.765773 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765673 2578 reconciler.go:29] "Reconciler: start to sync state" Jun 10 10:44:17.765915 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765902 2578 factory.go:153] Registering CRI-O factory Jun 10 10:44:17.765982 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.765920 2578 factory.go:223] Registration of the crio container factory successfully Jun 10 10:44:17.768755 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.768532 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jun 10 10:44:17.768874 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.765104 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-231.ec2.internal.18b7b2557519ed5b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-231.ec2.internal,UID:ip-10-0-138-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-231.ec2.internal,},FirstTimestamp:2026-06-10 10:44:17.754918235 +0000 UTC m=+0.403974198,LastTimestamp:2026-06-10 10:44:17.754918235 +0000 UTC m=+0.403974198,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-231.ec2.internal,}" Jun 10 10:44:17.768945 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.768927 2578 factory.go:103] Registering Raw factory Jun 10 10:44:17.768984 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.768946 2578 manager.go:1196] Started watching for new ooms in manager Jun 10 10:44:17.769581 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.769566 2578 manager.go:349] Starting recovery of all containers Jun 10 10:44:17.774448 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.774419 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jun 10 10:44:17.774560 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.774540 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jun 10 10:44:17.781371 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.781351 2578 manager.go:354] Recovery completed Jun 10 10:44:17.785308 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.785296 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:17.787197 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.787177 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:17.787278 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.787215 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:17.787278 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.787226 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:17.787777 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.787763 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jun 10 10:44:17.787777 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.787774 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jun 10 10:44:17.787902 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.787781 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:17.787902 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.787792 2578 state_mem.go:36] "Initialized new in-memory state store" Jun 10 10:44:17.788551 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.788532 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:17.788619 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.788566 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:17.788619 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.788581 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:17.789850 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.789838 2578 policy_none.go:49] "None policy: Start" Jun 10 10:44:17.789891 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.789855 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jun 10 10:44:17.789891 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.789867 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jun 10 10:44:17.791351 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.791340 2578 policy_none.go:47] "Start" Jun 10 10:44:17.791844 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.791745 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-231.ec2.internal.18b7b25577067895 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-231.ec2.internal,UID:ip-10-0-138-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-231.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-231.ec2.internal,},FirstTimestamp:2026-06-10 10:44:17.787197589 +0000 UTC m=+0.436253562,LastTimestamp:2026-06-10 10:44:17.787197589 +0000 UTC m=+0.436253562,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-231.ec2.internal,}" Jun 10 10:44:17.801639 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.801560 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-231.ec2.internal.18b7b2557706d1c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-231.ec2.internal,UID:ip-10-0-138-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-231.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-231.ec2.internal,},FirstTimestamp:2026-06-10 10:44:17.78722042 +0000 UTC m=+0.436276385,LastTimestamp:2026-06-10 10:44:17.78722042 +0000 UTC m=+0.436276385,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-231.ec2.internal,}" Jun 10 10:44:17.810912 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.810843 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-231.ec2.internal.18b7b2557706f6a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-231.ec2.internal,UID:ip-10-0-138-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-231.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-231.ec2.internal,},FirstTimestamp:2026-06-10 10:44:17.78722986 +0000 UTC m=+0.436285824,LastTimestamp:2026-06-10 10:44:17.78722986 +0000 UTC m=+0.436285824,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-231.ec2.internal,}" Jun 10 10:44:17.812527 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.812508 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4cnfn" Jun 10 10:44:17.822703 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.822636 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-231.ec2.internal.18b7b25577067895\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-231.ec2.internal.18b7b25577067895 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-231.ec2.internal,UID:ip-10-0-138-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-231.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-231.ec2.internal,},FirstTimestamp:2026-06-10 10:44:17.787197589 +0000 UTC m=+0.436253562,LastTimestamp:2026-06-10 10:44:17.788551832 +0000 UTC m=+0.437607802,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-231.ec2.internal,}" Jun 10 10:44:17.823885 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.823865 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4cnfn" Jun 10 10:44:17.827442 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.827428 2578 manager.go:337] "Starting Device Plugin manager" Jun 10 10:44:17.827518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.827466 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jun 10 10:44:17.827518 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.827476 2578 server.go:87] "Starting device plugin registration server" Jun 10 10:44:17.828155 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.828127 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jun 10 10:44:17.828269 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.828145 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jun 10 10:44:17.828366 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.828337 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jun 10 10:44:17.829134 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.828961 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jun 10 10:44:17.829134 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.828973 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jun 10 10:44:17.829134 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.829056 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jun 10 10:44:17.829312 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.829164 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:17.847029 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.846994 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jun 10 10:44:17.856716 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.848370 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jun 10 10:44:17.856716 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.848418 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jun 10 10:44:17.856716 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.848440 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jun 10 10:44:17.856716 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.848472 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jun 10 10:44:17.856716 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.850889 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:17.929372 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.929286 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:17.930498 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.930477 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:17.930564 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.930513 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:17.930564 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.930522 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:17.930564 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.930546 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.940594 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.940572 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.940661 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.940597 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-231.ec2.internal\": node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:17.949355 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.949319 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"] Jun 10 10:44:17.949476 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.949385 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:17.950292 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.950269 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:17.950380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.950300 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:17.950380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.950309 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:17.952858 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.952845 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:17.952951 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.952937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.953001 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.952985 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:17.953604 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.953583 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:17.953713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.953590 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:17.953713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.953635 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:17.953713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.953649 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:17.953713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.953613 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:17.953713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.953685 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:17.955724 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.955711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.955780 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.955736 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:17.956562 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.956546 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:17.956958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.956933 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:17.957065 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.956973 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:17.964987 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.964967 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:17.967109 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.967090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27887576b5ee87287ccccd6eae7bb07b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-231.ec2.internal\" (UID: \"27887576b5ee87287ccccd6eae7bb07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.967171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.967116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6846988ae79e098611519c1ed6b81389-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"6846988ae79e098611519c1ed6b81389\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.967171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:17.967134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6846988ae79e098611519c1ed6b81389-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"6846988ae79e098611519c1ed6b81389\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.972351 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.972335 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-231.ec2.internal\" not found" node="ip-10-0-138-231.ec2.internal" Jun 10 10:44:17.975698 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:17.975680 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-231.ec2.internal\" not found" node="ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.065630 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.065594 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.067834 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.067812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27887576b5ee87287ccccd6eae7bb07b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-231.ec2.internal\" (UID: \"27887576b5ee87287ccccd6eae7bb07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.067894 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.067849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6846988ae79e098611519c1ed6b81389-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"6846988ae79e098611519c1ed6b81389\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.067894 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.067860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27887576b5ee87287ccccd6eae7bb07b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-231.ec2.internal\" (UID: \"27887576b5ee87287ccccd6eae7bb07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.067894 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.067866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6846988ae79e098611519c1ed6b81389-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"6846988ae79e098611519c1ed6b81389\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.068001 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.067920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6846988ae79e098611519c1ed6b81389-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"6846988ae79e098611519c1ed6b81389\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.068001 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.067927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6846988ae79e098611519c1ed6b81389-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"6846988ae79e098611519c1ed6b81389\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.166483 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.166442 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.267238 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.267182 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.279497 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.279477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.282498 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.282473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.367631 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.367592 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.468149 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.468119 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.568767 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.568690 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.669226 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.669199 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.700529 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.700501 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jun 10 10:44:18.701228 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.700683 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 10 10:44:18.769741 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:18.769711 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found" Jun 10 10:44:18.818671 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.818640 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:18.821844 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:18.821761 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27887576b5ee87287ccccd6eae7bb07b.slice/crio-d3b4365b337d03970b5e81afb17199242a3ff540f8c7aec73329b61881eab2be WatchSource:0}: Error finding container d3b4365b337d03970b5e81afb17199242a3ff540f8c7aec73329b61881eab2be: Status 404 returned error can't find the container with id d3b4365b337d03970b5e81afb17199242a3ff540f8c7aec73329b61881eab2be Jun 10 10:44:18.822045 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:18.822026 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6846988ae79e098611519c1ed6b81389.slice/crio-08ca4223c94e925a3a967f6a7c8aa8d4bfa5f344573c0985000ffdd9c5ca9672 WatchSource:0}: Error finding container 08ca4223c94e925a3a967f6a7c8aa8d4bfa5f344573c0985000ffdd9c5ca9672: Status 404 returned error can't find the container with id 08ca4223c94e925a3a967f6a7c8aa8d4bfa5f344573c0985000ffdd9c5ca9672 Jun 10 10:44:18.826751 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.826733 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 10 10:44:18.827066 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.827020 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-09 10:39:17 +0000 UTC" deadline="2028-03-18 15:33:17.729284612 +0000 UTC" Jun 10 10:44:18.827066 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.827060 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15532h48m58.902228132s" Jun 10 10:44:18.853343 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.853287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" event={"ID":"6846988ae79e098611519c1ed6b81389","Type":"ContainerStarted","Data":"08ca4223c94e925a3a967f6a7c8aa8d4bfa5f344573c0985000ffdd9c5ca9672"} Jun 10 10:44:18.854257 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.854235 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" event={"ID":"27887576b5ee87287ccccd6eae7bb07b","Type":"ContainerStarted","Data":"d3b4365b337d03970b5e81afb17199242a3ff540f8c7aec73329b61881eab2be"} Jun 10 10:44:18.865403 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.865384 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.875361 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.875339 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 10 10:44:18.877407 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.877395 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" Jun 10 10:44:18.890820 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.890790 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 10 10:44:18.924092 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:18.924071 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:19.334320 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.334284 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:19.745542 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.745467 2578 apiserver.go:52] "Watching apiserver" Jun 10 10:44:19.750671 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.750642 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jun 10 10:44:19.751092 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.751062 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4lzjt","openshift-multus/multus-additional-cni-plugins-mh7hw","kube-system/konnectivity-agent-m86dg","kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal","openshift-multus/network-metrics-daemon-8t72g","openshift-network-diagnostics/network-check-target-ccsb6","openshift-network-operator/iptables-alerter-zwzdb","openshift-ovn-kubernetes/ovnkube-node-2s2n5","kube-system/global-pull-secret-syncer-ptw8k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d","openshift-cluster-node-tuning-operator/tuned-l8pk6","openshift-dns/node-resolver-29rcg","openshift-image-registry/node-ca-ps8th","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"] Jun 10 10:44:19.754060 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.754037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.756404 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.756093 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jun 10 10:44:19.756404 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.756130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.756404 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.756232 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jun 10 10:44:19.756404 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.756252 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w7lf2\"" Jun 10 10:44:19.756652 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.756482 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jun 10 10:44:19.757854 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.757834 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-khtsb\"" Jun 10 10:44:19.758096 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.758075 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jun 10 10:44:19.759963 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.758721 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jun 10 10:44:19.759963 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.758849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.760985 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.760772 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5hrp7\"" Jun 10 10:44:19.761922 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.761174 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jun 10 10:44:19.761922 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.761199 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jun 10 10:44:19.761922 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.761354 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jun 10 10:44:19.761922 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.761435 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jun 10 10:44:19.761922 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.761489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:19.761922 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.761557 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:19.763670 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.763581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:19.763718 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.763693 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:19.765489 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.765465 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jun 10 10:44:19.765585 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.765494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.767453 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.767432 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jun 10 10:44:19.767688 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.767658 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jun 10 10:44:19.767840 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.767656 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r2dzb\"" Jun 10 10:44:19.768067 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.767990 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jun 10 10:44:19.768537 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.768258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.770834 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770144 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jun 10 10:44:19.770834 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.770834 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.770257 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:19.770834 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770423 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hg2vt\"" Jun 10 10:44:19.770834 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770459 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jun 10 10:44:19.770834 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770617 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jun 10 10:44:19.770834 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770642 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jun 10 10:44:19.771190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770851 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jun 10 10:44:19.771190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.770862 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jun 10 10:44:19.772713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.772693 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:19.774308 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.774286 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r7np4\"" Jun 10 10:44:19.774423 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.774294 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jun 10 10:44:19.774556 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.774536 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jun 10 10:44:19.775202 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.775185 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.776192 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-kubelet\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.776703 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776417 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 10 10:44:19.776703 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-cni-netd\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.776840 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-lib-modules\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.776840 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xm9l\" (UniqueName: \"kubernetes.io/projected/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-kube-api-access-4xm9l\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.776840 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-socket-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.776840 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-netns\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777034 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-system-cni-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777034 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-cni-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777034 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776951 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jun 10 10:44:19.777034 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.776972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-cni-bin\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777034 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjx2\" (UniqueName: \"kubernetes.io/projected/5008bbcc-daab-4622-819f-bf83f17c50fe-kube-api-access-7vjx2\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777042 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-socket-dir-parent\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-run-netns\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovnkube-config\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-modprobe-d\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-log-socket\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.777316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-tuned\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-sys-fs\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777380 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-daemon-config\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777404 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s52c9\"" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16d98a46-dd9c-47d4-aa38-47e65d1a3342-iptables-alerter-script\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fxgp\" (UniqueName: \"kubernetes.io/projected/16d98a46-dd9c-47d4-aa38-47e65d1a3342-kube-api-access-5fxgp\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777506 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysconfig\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-device-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmv4v\" (UniqueName: \"kubernetes.io/projected/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-kube-api-access-rmv4v\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-etc-kubernetes\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-node-log\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-cni-bin\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.777669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-kubernetes\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-systemd\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-host\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-hostroot\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-conf-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-slash\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stc2\" (UniqueName: \"kubernetes.io/projected/30c39d51-86a4-424a-97f0-f9c2c568f8db-kube-api-access-8stc2\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-tmp\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9sjr\" (UniqueName: \"kubernetes.io/projected/8327523a-eb26-4c9d-847a-761b868b2d27-kube-api-access-m9sjr\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-kubelet\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-var-lib-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysctl-d\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.777994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5008bbcc-daab-4622-819f-bf83f17c50fe-cni-binary-copy\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-cni-multus\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-run\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-var-lib-kubelet\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-ovn\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778209 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-cnibin\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16d98a46-dd9c-47d4-aa38-47e65d1a3342-host-slash\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-systemd-units\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovnkube-script-lib\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-registration-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-systemd\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysctl-conf\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-etc-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-env-overrides\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovn-node-metrics-cert\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-sys\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-os-release\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.778969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-k8s-cni-cncf-io\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.779764 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.778599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-multus-certs\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.779764 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.779101 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jun 10 10:44:19.779764 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.779403 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lhc82\"" Jun 10 10:44:19.779764 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.779498 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jun 10 10:44:19.780075 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.780057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.782066 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.782049 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jun 10 10:44:19.782201 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.782075 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c44m8\"" Jun 10 10:44:19.782201 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.782097 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jun 10 10:44:19.782201 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.782051 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jun 10 10:44:19.798376 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.797984 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wr22x" Jun 10 10:44:19.801243 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.801012 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:19.806043 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.806024 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wr22x" Jun 10 10:44:19.866543 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.866508 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jun 10 10:44:19.879808 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-modprobe-d\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.879960 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-system-cni-dir\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.879960 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdzw\" (UniqueName: \"kubernetes.io/projected/6bffc05c-b337-483c-8b1f-907780f2d0d2-kube-api-access-shdzw\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.879960 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-log-socket\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.879960 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-tuned\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.879960 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-sys-fs\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.879960 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-daemon-config\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-modprobe-d\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f2862d8-beef-4218-b972-896923c4709a-dbus\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-sys-fs\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.879993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-log-socket\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16d98a46-dd9c-47d4-aa38-47e65d1a3342-iptables-alerter-script\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fxgp\" (UniqueName: \"kubernetes.io/projected/16d98a46-dd9c-47d4-aa38-47e65d1a3342-kube-api-access-5fxgp\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysconfig\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-device-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmv4v\" (UniqueName: \"kubernetes.io/projected/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-kube-api-access-rmv4v\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-etc-kubernetes\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8b8w\" (UniqueName: \"kubernetes.io/projected/12b531fb-fb79-47d1-adab-9eca526c52de-kube-api-access-n8b8w\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysconfig\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-device-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-node-log\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-cni-bin\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-node-log\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880281 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-cni-bin\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-kubernetes\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-kubernetes\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-systemd\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-etc-kubernetes\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-host\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-hostroot\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-systemd\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-conf-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a97c9c3b-0b82-410b-b415-b62f308b297b-agent-certs\") pod \"konnectivity-agent-m86dg\" (UID: \"a97c9c3b-0b82-410b-b415-b62f308b297b\") " pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-host\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-slash\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880532 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-hostroot\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-conf-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.880831 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-slash\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880564 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-daemon-config\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6wh\" (UniqueName: \"kubernetes.io/projected/29011c56-f7dc-4418-8881-1cdfc9f277f1-kube-api-access-9t6wh\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bffc05c-b337-483c-8b1f-907780f2d0d2-hosts-file\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16d98a46-dd9c-47d4-aa38-47e65d1a3342-iptables-alerter-script\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stc2\" (UniqueName: \"kubernetes.io/projected/30c39d51-86a4-424a-97f0-f9c2c568f8db-kube-api-access-8stc2\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-tmp\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9sjr\" (UniqueName: \"kubernetes.io/projected/8327523a-eb26-4c9d-847a-761b868b2d27-kube-api-access-m9sjr\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-kubelet\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-var-lib-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysctl-d\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5008bbcc-daab-4622-819f-bf83f17c50fe-cni-binary-copy\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-cni-multus\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-run\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-var-lib-kubelet\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-kubelet\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.881576 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-cni-multus\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-ovn\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-var-lib-kubelet\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysctl-d\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-ovn\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.880876 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-var-lib-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-cnibin\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881024 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b531fb-fb79-47d1-adab-9eca526c52de-host\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16d98a46-dd9c-47d4-aa38-47e65d1a3342-host-slash\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-cnibin\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-systemd-units\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16d98a46-dd9c-47d4-aa38-47e65d1a3342-host-slash\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovnkube-script-lib\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.882163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881121 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-systemd-units\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-run\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-registration-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-systemd\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysctl-conf\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-cnibin\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881362 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5008bbcc-daab-4622-819f-bf83f17c50fe-cni-binary-copy\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-etc-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-registration-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-etc-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-systemd\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-sysctl-conf\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-env-overrides\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovn-node-metrics-cert\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.883190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-sys\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-os-release\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-k8s-cni-cncf-io\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-multus-certs\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-kubelet\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-cni-netd\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881690 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-k8s-cni-cncf-io\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-os-release\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881724 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-kubelet\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-sys\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-lib-modules\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovnkube-script-lib\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881725 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-cni-netd\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xm9l\" (UniqueName: \"kubernetes.io/projected/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-kube-api-access-4xm9l\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-multus-certs\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-socket-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-env-overrides\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.884045 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881847 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-lib-modules\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-netns\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-os-release\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a97c9c3b-0b82-410b-b415-b62f308b297b-konnectivity-ca\") pod \"konnectivity-agent-m86dg\" (UID: \"a97c9c3b-0b82-410b-b415-b62f308b297b\") " pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-socket-dir\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.881997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-system-cni-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-run-netns\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-cni-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882063 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-system-cni-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-cni-bin\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjx2\" (UniqueName: \"kubernetes.io/projected/5008bbcc-daab-4622-819f-bf83f17c50fe-kube-api-access-7vjx2\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-cni-dir\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-socket-dir-parent\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-host-var-lib-cni-bin\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:19.884712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5008bbcc-daab-4622-819f-bf83f17c50fe-multus-socket-dir-parent\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882231 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f2862d8-beef-4218-b972-896923c4709a-kubelet-config\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bffc05c-b337-483c-8b1f-907780f2d0d2-tmp-dir\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b531fb-fb79-47d1-adab-9eca526c52de-serviceca\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.882312 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-run-openvswitch\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.882396 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:20.382363815 +0000 UTC m=+3.031419801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-run-netns\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovnkube-config\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882487 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c39d51-86a4-424a-97f0-f9c2c568f8db-host-run-netns\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.885431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.882960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovnkube-config\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.886031 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.886009 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-etc-tuned\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.886490 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.886282 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c39d51-86a4-424a-97f0-f9c2c568f8db-ovn-node-metrics-cert\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.887231 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.886987 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:44:19.887231 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.887009 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:44:19.887231 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.887018 2578 projected.go:196] Error preparing data for projected volume kube-api-access-q58t6 for pod openshift-network-diagnostics/network-check-target-ccsb6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:19.887231 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.887061 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6 podName:bc6fff07-0a5d-4065-8912-789624912795 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:20.387046289 +0000 UTC m=+3.036102244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q58t6" (UniqueName: "kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6") pod "network-check-target-ccsb6" (UID: "bc6fff07-0a5d-4065-8912-789624912795") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:19.888041 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.888018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-tmp\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.888916 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.888894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fxgp\" (UniqueName: \"kubernetes.io/projected/16d98a46-dd9c-47d4-aa38-47e65d1a3342-kube-api-access-5fxgp\") pod \"iptables-alerter-zwzdb\" (UID: \"16d98a46-dd9c-47d4-aa38-47e65d1a3342\") " pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:19.890440 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.890419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjx2\" (UniqueName: \"kubernetes.io/projected/5008bbcc-daab-4622-819f-bf83f17c50fe-kube-api-access-7vjx2\") pod \"multus-4lzjt\" (UID: \"5008bbcc-daab-4622-819f-bf83f17c50fe\") " pod="openshift-multus/multus-4lzjt" Jun 10 10:44:19.891339 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.891317 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stc2\" (UniqueName: \"kubernetes.io/projected/30c39d51-86a4-424a-97f0-f9c2c568f8db-kube-api-access-8stc2\") pod \"ovnkube-node-2s2n5\" (UID: \"30c39d51-86a4-424a-97f0-f9c2c568f8db\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:19.891670 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.891645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xm9l\" (UniqueName: \"kubernetes.io/projected/b73c7432-8e5b-49d3-a90b-57a8fdeccce8-kube-api-access-4xm9l\") pod \"tuned-l8pk6\" (UID: \"b73c7432-8e5b-49d3-a90b-57a8fdeccce8\") " pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:19.892158 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.892138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9sjr\" (UniqueName: \"kubernetes.io/projected/8327523a-eb26-4c9d-847a-761b868b2d27-kube-api-access-m9sjr\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:19.892430 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.892410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmv4v\" (UniqueName: \"kubernetes.io/projected/ca17ed57-d0dc-4b82-9628-3fd2b743ec2f-kube-api-access-rmv4v\") pod \"aws-ebs-csi-driver-node-fvm9d\" (UID: \"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:19.983206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-os-release\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.983206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a97c9c3b-0b82-410b-b415-b62f308b297b-konnectivity-ca\") pod \"konnectivity-agent-m86dg\" (UID: \"a97c9c3b-0b82-410b-b415-b62f308b297b\") " pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:19.983435 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.983435 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f2862d8-beef-4218-b972-896923c4709a-kubelet-config\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.983435 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bffc05c-b337-483c-8b1f-907780f2d0d2-tmp-dir\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.983435 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b531fb-fb79-47d1-adab-9eca526c52de-serviceca\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.983435 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983322 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-os-release\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.983435 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f2862d8-beef-4218-b972-896923c4709a-kubelet-config\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.983435 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.983418 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-system-cni-dir\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:19.983488 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret podName:7f2862d8-beef-4218-b972-896923c4709a nodeName:}" failed. No retries permitted until 2026-06-10 10:44:20.483467929 +0000 UTC m=+3.132523884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret") pod "global-pull-secret-syncer-ptw8k" (UID: "7f2862d8-beef-4218-b972-896923c4709a") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983507 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-system-cni-dir\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdzw\" (UniqueName: \"kubernetes.io/projected/6bffc05c-b337-483c-8b1f-907780f2d0d2-kube-api-access-shdzw\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f2862d8-beef-4218-b972-896923c4709a-dbus\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8b8w\" (UniqueName: \"kubernetes.io/projected/12b531fb-fb79-47d1-adab-9eca526c52de-kube-api-access-n8b8w\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a97c9c3b-0b82-410b-b415-b62f308b297b-agent-certs\") pod \"konnectivity-agent-m86dg\" (UID: \"a97c9c3b-0b82-410b-b415-b62f308b297b\") " pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983683 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bffc05c-b337-483c-8b1f-907780f2d0d2-tmp-dir\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.983845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6wh\" (UniqueName: \"kubernetes.io/projected/29011c56-f7dc-4418-8881-1cdfc9f277f1-kube-api-access-9t6wh\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bffc05c-b337-483c-8b1f-907780f2d0d2-hosts-file\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b531fb-fb79-47d1-adab-9eca526c52de-host\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-cnibin\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a97c9c3b-0b82-410b-b415-b62f308b297b-konnectivity-ca\") pod \"konnectivity-agent-m86dg\" (UID: \"a97c9c3b-0b82-410b-b415-b62f308b297b\") " pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bffc05c-b337-483c-8b1f-907780f2d0d2-hosts-file\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.983970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f2862d8-beef-4218-b972-896923c4709a-dbus\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.984045 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12b531fb-fb79-47d1-adab-9eca526c52de-host\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.984094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-cnibin\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.984185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29011c56-f7dc-4418-8881-1cdfc9f277f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984684 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.984320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984684 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.984406 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.984684 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.984476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12b531fb-fb79-47d1-adab-9eca526c52de-serviceca\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:19.984684 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.984476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29011c56-f7dc-4418-8881-1cdfc9f277f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:19.986357 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:19.986335 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a97c9c3b-0b82-410b-b415-b62f308b297b-agent-certs\") pod \"konnectivity-agent-m86dg\" (UID: \"a97c9c3b-0b82-410b-b415-b62f308b297b\") " pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:20.000293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.000221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6wh\" (UniqueName: \"kubernetes.io/projected/29011c56-f7dc-4418-8881-1cdfc9f277f1-kube-api-access-9t6wh\") pod \"multus-additional-cni-plugins-mh7hw\" (UID: \"29011c56-f7dc-4418-8881-1cdfc9f277f1\") " pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:20.000293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.000250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8b8w\" (UniqueName: \"kubernetes.io/projected/12b531fb-fb79-47d1-adab-9eca526c52de-kube-api-access-n8b8w\") pod \"node-ca-ps8th\" (UID: \"12b531fb-fb79-47d1-adab-9eca526c52de\") " pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:20.001502 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.001475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdzw\" (UniqueName: \"kubernetes.io/projected/6bffc05c-b337-483c-8b1f-907780f2d0d2-kube-api-access-shdzw\") pod \"node-resolver-29rcg\" (UID: \"6bffc05c-b337-483c-8b1f-907780f2d0d2\") " pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:20.068602 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.068561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" Jun 10 10:44:20.076138 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.076111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" Jun 10 10:44:20.088076 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.088048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lzjt" Jun 10 10:44:20.093467 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.093447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zwzdb" Jun 10 10:44:20.102287 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.102262 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:20.109068 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.109045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:20.119694 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.119663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" Jun 10 10:44:20.127439 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.127414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-29rcg" Jun 10 10:44:20.133231 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.133207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ps8th" Jun 10 10:44:20.388302 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.388207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:20.388302 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.388280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:20.388518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.388390 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:20.388518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.388392 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:44:20.388518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.388417 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:44:20.388518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.388429 2578 projected.go:196] Error preparing data for projected volume kube-api-access-q58t6 for pod openshift-network-diagnostics/network-check-target-ccsb6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:20.388518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.388450 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:21.388430099 +0000 UTC m=+4.037486053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:20.388518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.388465 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6 podName:bc6fff07-0a5d-4065-8912-789624912795 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:21.388455754 +0000 UTC m=+4.037511722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q58t6" (UniqueName: "kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6") pod "network-check-target-ccsb6" (UID: "bc6fff07-0a5d-4065-8912-789624912795") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:20.489534 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.489493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:20.489719 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.489644 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:20.489719 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.489708 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret podName:7f2862d8-beef-4218-b972-896923c4709a nodeName:}" failed. No retries permitted until 2026-06-10 10:44:21.489690871 +0000 UTC m=+4.138746822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret") pod "global-pull-secret-syncer-ptw8k" (UID: "7f2862d8-beef-4218-b972-896923c4709a") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:20.635853 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.635822 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca17ed57_d0dc_4b82_9628_3fd2b743ec2f.slice/crio-696729403d8197f779e27443c9fecb251cca5fc512abb1146f0879f2b7a92375 WatchSource:0}: Error finding container 696729403d8197f779e27443c9fecb251cca5fc512abb1146f0879f2b7a92375: Status 404 returned error can't find the container with id 696729403d8197f779e27443c9fecb251cca5fc512abb1146f0879f2b7a92375 Jun 10 10:44:20.655183 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.654942 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bffc05c_b337_483c_8b1f_907780f2d0d2.slice/crio-e15aac5b97151e92a9b3b8737fa9f22c0f9577256bdde89c7b1ede3cb7540865 WatchSource:0}: Error finding container e15aac5b97151e92a9b3b8737fa9f22c0f9577256bdde89c7b1ede3cb7540865: Status 404 returned error can't find the container with id e15aac5b97151e92a9b3b8737fa9f22c0f9577256bdde89c7b1ede3cb7540865 Jun 10 10:44:20.655866 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.655845 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5008bbcc_daab_4622_819f_bf83f17c50fe.slice/crio-76f4f2ad813fe5d5e1abac29305e3e1ecb0638feb24bed931e5c3acac058cf22 WatchSource:0}: Error finding container 76f4f2ad813fe5d5e1abac29305e3e1ecb0638feb24bed931e5c3acac058cf22: Status 404 returned error can't find the container with id 76f4f2ad813fe5d5e1abac29305e3e1ecb0638feb24bed931e5c3acac058cf22 Jun 10 10:44:20.662552 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.662528 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73c7432_8e5b_49d3_a90b_57a8fdeccce8.slice/crio-5151d4946927a7c0a29e85aebd078bfb6c3630f5150b0b4f2263138de6d568a8 WatchSource:0}: Error finding container 5151d4946927a7c0a29e85aebd078bfb6c3630f5150b0b4f2263138de6d568a8: Status 404 returned error can't find the container with id 5151d4946927a7c0a29e85aebd078bfb6c3630f5150b0b4f2263138de6d568a8 Jun 10 10:44:20.663464 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.663439 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c39d51_86a4_424a_97f0_f9c2c568f8db.slice/crio-8d827e99c1bde6f3034671da1956a7b0af8731def556cbce764a724855a30ae0 WatchSource:0}: Error finding container 8d827e99c1bde6f3034671da1956a7b0af8731def556cbce764a724855a30ae0: Status 404 returned error can't find the container with id 8d827e99c1bde6f3034671da1956a7b0af8731def556cbce764a724855a30ae0 Jun 10 10:44:20.665200 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.665152 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d98a46_dd9c_47d4_aa38_47e65d1a3342.slice/crio-6dcea2ce5ce2cb8bc86b4fe8aee690c7c24f95697565bbe4052d1e07a52748f4 WatchSource:0}: Error finding container 6dcea2ce5ce2cb8bc86b4fe8aee690c7c24f95697565bbe4052d1e07a52748f4: Status 404 returned error can't find the container with id 6dcea2ce5ce2cb8bc86b4fe8aee690c7c24f95697565bbe4052d1e07a52748f4 Jun 10 10:44:20.666743 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.666727 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97c9c3b_0b82_410b_b415_b62f308b297b.slice/crio-8fb94c62cda872ed0c35a45de42fd1bd43d7d85808c01adaa42f6fda700c8c47 WatchSource:0}: Error finding container 8fb94c62cda872ed0c35a45de42fd1bd43d7d85808c01adaa42f6fda700c8c47: Status 404 returned error can't find the container with id 8fb94c62cda872ed0c35a45de42fd1bd43d7d85808c01adaa42f6fda700c8c47 Jun 10 10:44:20.668566 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.668465 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29011c56_f7dc_4418_8881_1cdfc9f277f1.slice/crio-ccaee8ac4dc8bd8241b992218d40187a5ae88ef22cec820ec66ec16c583488ed WatchSource:0}: Error finding container ccaee8ac4dc8bd8241b992218d40187a5ae88ef22cec820ec66ec16c583488ed: Status 404 returned error can't find the container with id ccaee8ac4dc8bd8241b992218d40187a5ae88ef22cec820ec66ec16c583488ed Jun 10 10:44:20.669791 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:20.669750 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b531fb_fb79_47d1_adab_9eca526c52de.slice/crio-0af6dda1a934722a6dccd17bb51b7e51cc0b89251d69692bf85fb209c99d2909 WatchSource:0}: Error finding container 0af6dda1a934722a6dccd17bb51b7e51cc0b89251d69692bf85fb209c99d2909: Status 404 returned error can't find the container with id 0af6dda1a934722a6dccd17bb51b7e51cc0b89251d69692bf85fb209c99d2909 Jun 10 10:44:20.809823 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.809746 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-09 10:39:19 +0000 UTC" deadline="2027-11-04 22:51:24.819121626 +0000 UTC" Jun 10 10:44:20.809823 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.809813 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12300h7m4.009312523s" Jun 10 10:44:20.849206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.849179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:20.849373 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:20.849297 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:20.858312 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.858276 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerStarted","Data":"ccaee8ac4dc8bd8241b992218d40187a5ae88ef22cec820ec66ec16c583488ed"} Jun 10 10:44:20.859429 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.859398 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m86dg" event={"ID":"a97c9c3b-0b82-410b-b415-b62f308b297b","Type":"ContainerStarted","Data":"8fb94c62cda872ed0c35a45de42fd1bd43d7d85808c01adaa42f6fda700c8c47"} Jun 10 10:44:20.862024 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.861200 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zwzdb" event={"ID":"16d98a46-dd9c-47d4-aa38-47e65d1a3342","Type":"ContainerStarted","Data":"6dcea2ce5ce2cb8bc86b4fe8aee690c7c24f95697565bbe4052d1e07a52748f4"} Jun 10 10:44:20.862952 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.862871 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"8d827e99c1bde6f3034671da1956a7b0af8731def556cbce764a724855a30ae0"} Jun 10 10:44:20.864273 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.864253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" event={"ID":"b73c7432-8e5b-49d3-a90b-57a8fdeccce8","Type":"ContainerStarted","Data":"5151d4946927a7c0a29e85aebd078bfb6c3630f5150b0b4f2263138de6d568a8"} Jun 10 10:44:20.865298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.865275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" event={"ID":"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f","Type":"ContainerStarted","Data":"696729403d8197f779e27443c9fecb251cca5fc512abb1146f0879f2b7a92375"} Jun 10 10:44:20.866314 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.866296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lzjt" event={"ID":"5008bbcc-daab-4622-819f-bf83f17c50fe","Type":"ContainerStarted","Data":"76f4f2ad813fe5d5e1abac29305e3e1ecb0638feb24bed931e5c3acac058cf22"} Jun 10 10:44:20.867321 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.867300 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-29rcg" event={"ID":"6bffc05c-b337-483c-8b1f-907780f2d0d2","Type":"ContainerStarted","Data":"e15aac5b97151e92a9b3b8737fa9f22c0f9577256bdde89c7b1ede3cb7540865"} Jun 10 10:44:20.869323 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.869302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" event={"ID":"27887576b5ee87287ccccd6eae7bb07b","Type":"ContainerStarted","Data":"399e65b2a683b5b51c185abf7f59e8b0c28f576b398e38db292ef13eb13949f1"} Jun 10 10:44:20.870479 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.870458 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ps8th" event={"ID":"12b531fb-fb79-47d1-adab-9eca526c52de","Type":"ContainerStarted","Data":"0af6dda1a934722a6dccd17bb51b7e51cc0b89251d69692bf85fb209c99d2909"} Jun 10 10:44:20.885357 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:20.885299 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" podStartSLOduration=2.885282208 podStartE2EDuration="2.885282208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:20.884921163 +0000 UTC m=+3.533977135" watchObservedRunningTime="2026-06-10 10:44:20.885282208 +0000 UTC m=+3.534338168" Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.396069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.396147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.396287 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.396352 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:23.396331503 +0000 UTC m=+6.045387456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.396823 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.396847 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.396861 2578 projected.go:196] Error preparing data for projected volume kube-api-access-q58t6 for pod openshift-network-diagnostics/network-check-target-ccsb6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:21.396992 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.396908 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6 podName:bc6fff07-0a5d-4065-8912-789624912795 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:23.396891784 +0000 UTC m=+6.045947738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q58t6" (UniqueName: "kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6") pod "network-check-target-ccsb6" (UID: "bc6fff07-0a5d-4065-8912-789624912795") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:21.497541 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.497502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:21.497720 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.497692 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:21.497775 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.497759 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret podName:7f2862d8-beef-4218-b972-896923c4709a nodeName:}" failed. No retries permitted until 2026-06-10 10:44:23.497739773 +0000 UTC m=+6.146795741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret") pod "global-pull-secret-syncer-ptw8k" (UID: "7f2862d8-beef-4218-b972-896923c4709a") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:21.810614 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.810473 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-09 10:39:19 +0000 UTC" deadline="2028-01-01 19:13:46.641373062 +0000 UTC" Jun 10 10:44:21.810614 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.810535 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13688h29m24.830842368s" Jun 10 10:44:21.851917 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.851055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:21.851917 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.851170 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:21.851917 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.851238 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:21.851917 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:21.851309 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:21.897693 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.897654 2578 generic.go:356] "Generic (PLEG): container finished" podID="6846988ae79e098611519c1ed6b81389" containerID="d6bae6fa2e4bff5551a03506949a2fde8f5a9b56128602253f3918dba972e583" exitCode=0 Jun 10 10:44:21.898626 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:21.898598 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" event={"ID":"6846988ae79e098611519c1ed6b81389","Type":"ContainerDied","Data":"d6bae6fa2e4bff5551a03506949a2fde8f5a9b56128602253f3918dba972e583"} Jun 10 10:44:22.849719 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:22.849682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:22.850191 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:22.849862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:22.921476 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:22.921438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" event={"ID":"6846988ae79e098611519c1ed6b81389","Type":"ContainerStarted","Data":"545b6bed3cfff594ef6c26e3b25796cb9004a822f912c072fa42bdd11f3bb092"} Jun 10 10:44:23.415619 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:23.415581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:23.415791 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:23.415666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:23.415899 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.415851 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:23.415955 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.415914 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:27.415896204 +0000 UTC m=+10.064952161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:23.416345 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.416325 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:44:23.416417 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.416350 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:44:23.416417 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.416364 2578 projected.go:196] Error preparing data for projected volume kube-api-access-q58t6 for pod openshift-network-diagnostics/network-check-target-ccsb6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:23.416417 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.416410 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6 podName:bc6fff07-0a5d-4065-8912-789624912795 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:27.416394494 +0000 UTC m=+10.065450449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q58t6" (UniqueName: "kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6") pod "network-check-target-ccsb6" (UID: "bc6fff07-0a5d-4065-8912-789624912795") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:23.516471 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:23.516431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:23.516652 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.516608 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:23.516717 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.516669 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret podName:7f2862d8-beef-4218-b972-896923c4709a nodeName:}" failed. No retries permitted until 2026-06-10 10:44:27.516652445 +0000 UTC m=+10.165708403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret") pod "global-pull-secret-syncer-ptw8k" (UID: "7f2862d8-beef-4218-b972-896923c4709a") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:23.849658 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:23.849576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:23.849820 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.849717 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:23.850269 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:23.850249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:23.850378 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:23.850359 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:24.848980 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:24.848937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:24.849129 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:24.849079 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:25.849247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:25.849204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:25.849247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:25.849235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:25.849745 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:25.849336 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:25.849745 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:25.849458 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:26.848888 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:26.848855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:26.849059 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:26.849037 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:27.450869 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:27.450832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:27.451354 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:27.450914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:27.451354 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.451042 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:44:27.451354 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.451063 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:27.451354 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.451070 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:44:27.451354 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.451090 2578 projected.go:196] Error preparing data for projected volume kube-api-access-q58t6 for pod openshift-network-diagnostics/network-check-target-ccsb6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:27.451354 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.451130 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:35.451109853 +0000 UTC m=+18.100165823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:27.451354 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.451150 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6 podName:bc6fff07-0a5d-4065-8912-789624912795 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:35.451140616 +0000 UTC m=+18.100196582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q58t6" (UniqueName: "kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6") pod "network-check-target-ccsb6" (UID: "bc6fff07-0a5d-4065-8912-789624912795") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:27.551874 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:27.551845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:27.552057 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.552000 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:27.552057 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.552054 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret podName:7f2862d8-beef-4218-b972-896923c4709a nodeName:}" failed. No retries permitted until 2026-06-10 10:44:35.552037898 +0000 UTC m=+18.201093849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret") pod "global-pull-secret-syncer-ptw8k" (UID: "7f2862d8-beef-4218-b972-896923c4709a") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:27.849669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:27.849594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:27.849832 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.849696 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:27.850232 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:27.850214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:27.850326 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:27.850307 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:28.849014 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:28.848976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:28.849401 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:28.849132 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:29.849724 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:29.849339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:29.849724 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:29.849449 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:29.849724 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:29.849517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:29.849724 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:29.849621 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:30.849069 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:30.849027 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:30.849231 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:30.849175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:31.852485 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:31.852451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:31.852959 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:31.852574 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:31.852959 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:31.852625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:31.852959 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:31.852716 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:32.849619 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:32.849584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:32.849792 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:32.849721 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:33.848780 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:33.848750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:33.849171 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:33.848872 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:33.849171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:33.848925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:33.849171 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:33.849022 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:34.848762 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:34.848729 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:34.848957 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:34.848865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:35.508636 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:35.508594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:35.508837 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:35.508669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:35.508837 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.508789 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:35.508837 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.508820 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:44:35.508837 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.508838 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:44:35.509069 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.508851 2578 projected.go:196] Error preparing data for projected volume kube-api-access-q58t6 for pod openshift-network-diagnostics/network-check-target-ccsb6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:35.509069 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.508891 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:51.508866444 +0000 UTC m=+34.157922400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:35.509069 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.508912 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6 podName:bc6fff07-0a5d-4065-8912-789624912795 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:51.508903489 +0000 UTC m=+34.157959440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q58t6" (UniqueName: "kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6") pod "network-check-target-ccsb6" (UID: "bc6fff07-0a5d-4065-8912-789624912795") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:35.609517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:35.609476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:35.609698 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.609588 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:35.609698 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.609651 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret podName:7f2862d8-beef-4218-b972-896923c4709a nodeName:}" failed. No retries permitted until 2026-06-10 10:44:51.609632164 +0000 UTC m=+34.258688119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret") pod "global-pull-secret-syncer-ptw8k" (UID: "7f2862d8-beef-4218-b972-896923c4709a") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:35.848962 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:35.848880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:35.849356 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:35.848887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:35.849356 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.849015 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:35.849356 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:35.849124 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:36.848665 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:36.848623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:36.848958 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:36.848768 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:37.855619 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:37.855583 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:37.856052 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:37.855683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:37.856052 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:37.855760 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:37.856052 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:37.855807 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:38.848706 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:38.848670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:38.849773 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:38.848817 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:39.854515 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.854494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:39.855120 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:39.854587 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:39.855120 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.854667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:39.855120 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:39.854750 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:39.959306 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.959283 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:44:39.959714 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.959690 2578 generic.go:356] "Generic (PLEG): container finished" podID="30c39d51-86a4-424a-97f0-f9c2c568f8db" containerID="dced9fb45a9307e4f0c4ef4edc38dc8b4b1190f04dc32923e13f38088b2961be" exitCode=1 Jun 10 10:44:39.959787 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.959761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerDied","Data":"dced9fb45a9307e4f0c4ef4edc38dc8b4b1190f04dc32923e13f38088b2961be"} Jun 10 10:44:39.959875 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.959856 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"18e62bd52a57979744e37c2f894430e16803450334185f6baf8d311c60f63bf4"} Jun 10 10:44:39.961238 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.961215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" event={"ID":"b73c7432-8e5b-49d3-a90b-57a8fdeccce8","Type":"ContainerStarted","Data":"3b10715e89a087f1cd574b8b42620b576d77b243bf8b9f195679ba18f095918b"} Jun 10 10:44:39.962713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.962691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" event={"ID":"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f","Type":"ContainerStarted","Data":"0e947354cf4ead92c680ff899c883d7896fa0c386bd3982c5641f00a71978595"} Jun 10 10:44:39.964308 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.964278 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lzjt" event={"ID":"5008bbcc-daab-4622-819f-bf83f17c50fe","Type":"ContainerStarted","Data":"41a6cb1c8557e9de59a43f7ecc5fc7bf5957849c6188dc3a41535cf388bbbfb4"} Jun 10 10:44:39.965825 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.965784 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-29rcg" event={"ID":"6bffc05c-b337-483c-8b1f-907780f2d0d2","Type":"ContainerStarted","Data":"6fec2bb57094c71cf7ce442dc54df28156370467c62ec74ee5ef56e19f74ebba"} Jun 10 10:44:39.967159 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.967131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ps8th" event={"ID":"12b531fb-fb79-47d1-adab-9eca526c52de","Type":"ContainerStarted","Data":"9c1b2c88fdc2dea567b970fe4807b8c24b651695f4d487f62250d83a275b750e"} Jun 10 10:44:39.968839 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.968790 2578 generic.go:356] "Generic (PLEG): container finished" podID="29011c56-f7dc-4418-8881-1cdfc9f277f1" containerID="d326500ee948903638c74fab207eeb4c3bea8aa6592ba864e12f916cff08bc3d" exitCode=0 Jun 10 10:44:39.968920 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.968842 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerDied","Data":"d326500ee948903638c74fab207eeb4c3bea8aa6592ba864e12f916cff08bc3d"} Jun 10 10:44:39.973020 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.972998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m86dg" event={"ID":"a97c9c3b-0b82-410b-b415-b62f308b297b","Type":"ContainerStarted","Data":"36e3d693434f7e48802b45f1aa1ca0986b39a558521e25db70c7cfb4345044c2"} Jun 10 10:44:39.977353 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.977308 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" podStartSLOduration=21.977295809 podStartE2EDuration="21.977295809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:22.939181732 +0000 UTC m=+5.588237705" watchObservedRunningTime="2026-06-10 10:44:39.977295809 +0000 UTC m=+22.626351785" Jun 10 10:44:39.977849 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:39.977825 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l8pk6" podStartSLOduration=4.109727602 podStartE2EDuration="22.977792824s" totalImagesPullingTime="18.868065222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:39.977212479 +0000 UTC m=+22.626268465" watchObservedRunningTime="2026-06-10 10:44:39.977792824 +0000 UTC m=+22.626848796" Jun 10 10:44:40.019482 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.019434 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m86dg" podStartSLOduration=3.181090526 podStartE2EDuration="22.019418152s" totalImagesPullingTime="18.838327626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:39.991919621 +0000 UTC m=+22.640975595" watchObservedRunningTime="2026-06-10 10:44:40.019418152 +0000 UTC m=+22.668474125" Jun 10 10:44:40.050272 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.050233 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-29rcg" podStartSLOduration=3.4513249950000002 podStartE2EDuration="22.050218498s" totalImagesPullingTime="18.598893503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:40.035326515 +0000 UTC m=+22.684382515" watchObservedRunningTime="2026-06-10 10:44:40.050218498 +0000 UTC m=+22.699274470" Jun 10 10:44:40.050393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.050301 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ps8th" podStartSLOduration=3.190548293 podStartE2EDuration="22.050296607s" totalImagesPullingTime="18.859748314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:40.050120728 +0000 UTC m=+22.699176701" watchObservedRunningTime="2026-06-10 10:44:40.050296607 +0000 UTC m=+22.699352577" Jun 10 10:44:40.849786 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.849498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:40.849977 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:40.849887 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:40.979182 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.979155 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:44:40.980101 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.980073 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"2f267757921f5246c3501ce6468446b5a7d407564420e04afc87cca6b0b70369"} Jun 10 10:44:40.980212 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.980137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"8905c6cdedab92d81774f9e1d25cb1a2b107e5358cd3616d9ff28a7a77d22655"} Jun 10 10:44:40.980212 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.980179 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"a4dae7e3bbd1600c56991dfae8f72bc1512348b8ac2f1655293ab03c429860e2"} Jun 10 10:44:40.980212 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:40.980211 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"7925cce5f11277b55837accee7e665ec8aebd08ff09b2d59fe7da39ac6110b0a"} Jun 10 10:44:41.208593 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.208556 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jun 10 10:44:41.843319 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.843204 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-10T10:44:41.208575365Z","UUID":"ce1bb843-6064-4cfd-abb6-95e04a58a037","Handler":null,"Name":"","Endpoint":""} Jun 10 10:44:41.844964 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.844882 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jun 10 10:44:41.844964 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.844929 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jun 10 10:44:41.849157 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.849137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:41.849656 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:41.849324 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:41.849857 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.849775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:41.849928 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:41.849912 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:41.984899 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.984394 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zwzdb" event={"ID":"16d98a46-dd9c-47d4-aa38-47e65d1a3342","Type":"ContainerStarted","Data":"585374ef8402d99ba5e3967f4e786397a5d4606cf24227bd9ee7bd6475ae9ec2"} Jun 10 10:44:41.986596 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:41.986553 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" event={"ID":"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f","Type":"ContainerStarted","Data":"1cfbc4ee140eb957e3dad4fe9b0a3957510ee0dda8edf10adbe58b840d04cdc9"} Jun 10 10:44:42.001900 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:42.001848 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4lzjt" podStartSLOduration=5.098135756 podStartE2EDuration="24.001829059s" totalImagesPullingTime="18.903693303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:40.067001768 +0000 UTC m=+22.716057751" watchObservedRunningTime="2026-06-10 10:44:42.001829059 +0000 UTC m=+24.650885032" Jun 10 10:44:42.849051 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:42.848872 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:42.849223 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:42.849154 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:42.990919 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:42.990882 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" event={"ID":"ca17ed57-d0dc-4b82-9628-3fd2b743ec2f","Type":"ContainerStarted","Data":"ee1fbd6768e23dfd055ed9d974b5495fd1f655fb808c996a3e03abea3b0b2c68"} Jun 10 10:44:43.009385 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.009337 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zwzdb" podStartSLOduration=10.73595512 podStartE2EDuration="25.009318689s" totalImagesPullingTime="14.273363569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:42.002524877 +0000 UTC m=+24.651580831" watchObservedRunningTime="2026-06-10 10:44:43.009318689 +0000 UTC m=+25.658374656" Jun 10 10:44:43.009666 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.009639 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvm9d" podStartSLOduration=5.065041236 podStartE2EDuration="26.00962925s" totalImagesPullingTime="20.944588014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:43.00941225 +0000 UTC m=+25.658468224" watchObservedRunningTime="2026-06-10 10:44:43.00962925 +0000 UTC m=+25.658685223" Jun 10 10:44:43.700904 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.700866 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:43.701609 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.701587 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:43.849367 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.849340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:43.849512 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.849346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:43.849512 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:43.849449 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:43.849615 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:43.849557 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:43.995840 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.995817 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:44:43.997969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.996482 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:43.997969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:43.996962 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m86dg" Jun 10 10:44:44.848927 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:44.848897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:44.849066 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:44.849000 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:45.000123 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:45.000092 2578 generic.go:356] "Generic (PLEG): container finished" podID="29011c56-f7dc-4418-8881-1cdfc9f277f1" containerID="8b83e6c18c535f25cb5fd11629ac687ef1589356c490ada3459242fa7c768fb7" exitCode=0 Jun 10 10:44:45.000648 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:45.000183 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerDied","Data":"8b83e6c18c535f25cb5fd11629ac687ef1589356c490ada3459242fa7c768fb7"} Jun 10 10:44:45.002989 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:45.002969 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:44:45.003377 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:45.003356 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"95ed446aa6d9239a0568b6b7047d9babb6e33b433e3cb839866465d2ec151c60"} Jun 10 10:44:45.848937 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:45.848895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:45.849091 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:45.848895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:45.849091 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:45.849033 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:45.849091 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:45.849074 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:46.007688 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:46.007609 2578 generic.go:356] "Generic (PLEG): container finished" podID="29011c56-f7dc-4418-8881-1cdfc9f277f1" containerID="08c7ceda5be8877019f5f1b9a8b83aadd53a80f297b4e08759f99fdab9799dd0" exitCode=0 Jun 10 10:44:46.008171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:46.007702 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerDied","Data":"08c7ceda5be8877019f5f1b9a8b83aadd53a80f297b4e08759f99fdab9799dd0"} Jun 10 10:44:46.849413 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:46.849190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:46.849555 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:46.849446 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:47.011718 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.011686 2578 generic.go:356] "Generic (PLEG): container finished" podID="29011c56-f7dc-4418-8881-1cdfc9f277f1" containerID="3d5cfc6fa47b6d40b77b7ba3dd0903dca43aee5c20e507b610d71ce8b7306aff" exitCode=0 Jun 10 10:44:47.012205 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.011778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerDied","Data":"3d5cfc6fa47b6d40b77b7ba3dd0903dca43aee5c20e507b610d71ce8b7306aff"} Jun 10 10:44:47.015546 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.015527 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:44:47.015939 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.015916 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"0be2bc3f56ec13e04410ba9ed62811e775e13df8a518bd7367fdf3c37b3eb5da"} Jun 10 10:44:47.016199 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.016181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:47.016334 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.016320 2578 scope.go:117] "RemoveContainer" containerID="dced9fb45a9307e4f0c4ef4edc38dc8b4b1190f04dc32923e13f38088b2961be" Jun 10 10:44:47.032700 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.032682 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:47.850123 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.850091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:47.850291 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:47.850208 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:47.850291 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:47.850268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:47.850405 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:47.850384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:48.027059 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.027034 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:44:48.027497 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.027465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" event={"ID":"30c39d51-86a4-424a-97f0-f9c2c568f8db","Type":"ContainerStarted","Data":"8b00796eb1d954b9974f212f91199c2ddc1df2782ebc9d6c0aff14cb26dcddeb"} Jun 10 10:44:48.028604 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.028585 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:48.028713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.028647 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:48.049887 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.049858 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:44:48.061080 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.060368 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" podStartSLOduration=11.13787952 podStartE2EDuration="30.060348495s" totalImagesPullingTime="18.922468975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:48.059292565 +0000 UTC m=+30.708348539" watchObservedRunningTime="2026-06-10 10:44:48.060348495 +0000 UTC m=+30.709404469" Jun 10 10:44:48.178762 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.178731 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8t72g"] Jun 10 10:44:48.178930 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.178890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:48.179034 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:48.179005 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:48.183843 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.183511 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccsb6"] Jun 10 10:44:48.183843 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.183675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:48.183843 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:48.183777 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:48.184144 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.184121 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ptw8k"] Jun 10 10:44:48.184270 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:48.184223 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:48.185731 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:48.184319 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:49.849461 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:49.849252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:49.849885 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:49.849252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:49.849885 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:49.849575 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:44:49.849885 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:49.849649 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccsb6" podUID="bc6fff07-0a5d-4065-8912-789624912795" Jun 10 10:44:49.849885 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:49.849279 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:49.849885 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:49.849745 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ptw8k" podUID="7f2862d8-beef-4218-b972-896923c4709a" Jun 10 10:44:51.528264 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.528226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:51.529055 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.528289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:51.529055 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.528397 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:51.529055 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.528416 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:44:51.529055 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.528438 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:44:51.529055 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.528453 2578 projected.go:196] Error preparing data for projected volume kube-api-access-q58t6 for pod openshift-network-diagnostics/network-check-target-ccsb6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:51.529055 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.528461 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:23.528443757 +0000 UTC m=+66.177499723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:44:51.529055 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.528525 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6 podName:bc6fff07-0a5d-4065-8912-789624912795 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:23.528508382 +0000 UTC m=+66.177564334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q58t6" (UniqueName: "kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6") pod "network-check-target-ccsb6" (UID: "bc6fff07-0a5d-4065-8912-789624912795") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:44:51.629525 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.629489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:51.629726 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.629657 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:51.629825 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.629740 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret podName:7f2862d8-beef-4218-b972-896923c4709a nodeName:}" failed. No retries permitted until 2026-06-10 10:45:23.629718901 +0000 UTC m=+66.278774852 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret") pod "global-pull-secret-syncer-ptw8k" (UID: "7f2862d8-beef-4218-b972-896923c4709a") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:44:51.646457 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.646429 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeReady" Jun 10 10:44:51.646633 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.646585 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jun 10 10:44:51.682364 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.682282 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b6f49bc56-wpzlc"] Jun 10 10:44:51.698881 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.698853 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vg7kl"] Jun 10 10:44:51.699021 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.698973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.707246 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.707220 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jun 10 10:44:51.707379 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.707248 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jun 10 10:44:51.707379 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.707308 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jun 10 10:44:51.707814 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.707774 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6lbht\"" Jun 10 10:44:51.713280 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.713265 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jun 10 10:44:51.725035 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.725008 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b6f49bc56-wpzlc"] Jun 10 10:44:51.725161 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.725043 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-42gfm"] Jun 10 10:44:51.725161 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.725129 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.727997 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.727612 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-977dh\"" Jun 10 10:44:51.727997 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.727764 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jun 10 10:44:51.727997 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.727787 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jun 10 10:44:51.745013 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.744975 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42gfm"] Jun 10 10:44:51.745013 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.745013 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vg7kl"] Jun 10 10:44:51.745175 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.745087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:51.747065 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.747036 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jun 10 10:44:51.747177 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.747103 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jun 10 10:44:51.747177 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.747109 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dkm7w\"" Jun 10 10:44:51.747177 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.747123 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jun 10 10:44:51.830369 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.830369 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-certificates\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd2e06-fcd6-43b0-aff9-6f114eddd501-tmp-dir\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-image-registry-private-configuration\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmsj\" (UniqueName: \"kubernetes.io/projected/125476fa-5f69-45a5-8232-31541c6be892-kube-api-access-sdmsj\") pod \"ingress-canary-42gfm\" (UID: \"125476fa-5f69-45a5-8232-31541c6be892\") " pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cacd2e06-fcd6-43b0-aff9-6f114eddd501-config-volume\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwds\" (UniqueName: \"kubernetes.io/projected/cacd2e06-fcd6-43b0-aff9-6f114eddd501-kube-api-access-ngwds\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125476fa-5f69-45a5-8232-31541c6be892-cert\") pod \"ingress-canary-42gfm\" (UID: \"125476fa-5f69-45a5-8232-31541c6be892\") " pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.830643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3da4c40-3de7-4f9b-80c5-08d9042085e8-ca-trust-extracted\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.830942 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-trusted-ca\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.830942 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-installation-pull-secrets\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.830942 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-bound-sa-token\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.830942 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.830737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2dg\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-kube-api-access-rk2dg\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.849738 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.849707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:44:51.849842 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.849754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:44:51.849842 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.849710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:44:51.852744 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.852709 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 10 10:44:51.852869 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.852716 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 10 10:44:51.852869 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.852778 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgxv7\"" Jun 10 10:44:51.852869 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.852720 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 10 10:44:51.852992 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.852781 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 10 10:44:51.852992 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.852720 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8hbqv\"" Jun 10 10:44:51.931468 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.931468 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-certificates\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.931687 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd2e06-fcd6-43b0-aff9-6f114eddd501-tmp-dir\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.931687 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-image-registry-private-configuration\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.931687 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.931604 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:44:51.931687 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.931665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:52.431650038 +0000 UTC m=+35.080705989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:44:51.931687 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmsj\" (UniqueName: \"kubernetes.io/projected/125476fa-5f69-45a5-8232-31541c6be892-kube-api-access-sdmsj\") pod \"ingress-canary-42gfm\" (UID: \"125476fa-5f69-45a5-8232-31541c6be892\") " pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:51.931965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cacd2e06-fcd6-43b0-aff9-6f114eddd501-config-volume\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.931965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwds\" (UniqueName: \"kubernetes.io/projected/cacd2e06-fcd6-43b0-aff9-6f114eddd501-kube-api-access-ngwds\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.931965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.931912 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd2e06-fcd6-43b0-aff9-6f114eddd501-tmp-dir\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.932179 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125476fa-5f69-45a5-8232-31541c6be892-cert\") pod \"ingress-canary-42gfm\" (UID: \"125476fa-5f69-45a5-8232-31541c6be892\") " pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:51.932179 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.932179 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3da4c40-3de7-4f9b-80c5-08d9042085e8-ca-trust-extracted\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.932179 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932113 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-certificates\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.932179 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.932177 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.932194 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-trusted-ca\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:51.932243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:52.432224939 +0000 UTC m=+35.081280905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-installation-pull-secrets\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cacd2e06-fcd6-43b0-aff9-6f114eddd501-config-volume\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-bound-sa-token\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2dg\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-kube-api-access-rk2dg\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.932438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.932386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3da4c40-3de7-4f9b-80c5-08d9042085e8-ca-trust-extracted\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.933108 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.933086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-trusted-ca\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.936097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.936071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-image-registry-private-configuration\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.936097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.936091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125476fa-5f69-45a5-8232-31541c6be892-cert\") pod \"ingress-canary-42gfm\" (UID: \"125476fa-5f69-45a5-8232-31541c6be892\") " pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:51.936219 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.936135 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-installation-pull-secrets\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.940733 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.940711 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-bound-sa-token\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:51.940849 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.940714 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwds\" (UniqueName: \"kubernetes.io/projected/cacd2e06-fcd6-43b0-aff9-6f114eddd501-kube-api-access-ngwds\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:51.941296 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.941271 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmsj\" (UniqueName: \"kubernetes.io/projected/125476fa-5f69-45a5-8232-31541c6be892-kube-api-access-sdmsj\") pod \"ingress-canary-42gfm\" (UID: \"125476fa-5f69-45a5-8232-31541c6be892\") " pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:51.941427 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:51.941407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2dg\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-kube-api-access-rk2dg\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:52.040319 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:52.040279 2578 generic.go:356] "Generic (PLEG): container finished" podID="29011c56-f7dc-4418-8881-1cdfc9f277f1" containerID="04620deaff9573e773a34f9d9811f4da4872542da288f5f946fa694d7b9bfbf2" exitCode=0 Jun 10 10:44:52.040476 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:52.040336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerDied","Data":"04620deaff9573e773a34f9d9811f4da4872542da288f5f946fa694d7b9bfbf2"} Jun 10 10:44:52.056371 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:52.056344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42gfm" Jun 10 10:44:52.276755 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:52.276513 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42gfm"] Jun 10 10:44:52.309720 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:44:52.309687 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125476fa_5f69_45a5_8232_31541c6be892.slice/crio-ce628019c2442f98b5f1861ccba3d8184252f232e4cdc4393b61ec480af5fa1b WatchSource:0}: Error finding container ce628019c2442f98b5f1861ccba3d8184252f232e4cdc4393b61ec480af5fa1b: Status 404 returned error can't find the container with id ce628019c2442f98b5f1861ccba3d8184252f232e4cdc4393b61ec480af5fa1b Jun 10 10:44:52.436665 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:52.436624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:52.436853 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:52.436691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:52.436853 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:52.436768 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:44:52.436853 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:52.436786 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:44:52.436853 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:52.436826 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:44:52.436985 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:52.436866 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:53.436846703 +0000 UTC m=+36.085902674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:44:52.436985 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:52.436880 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:53.436873921 +0000 UTC m=+36.085929872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:44:53.046024 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:53.045985 2578 generic.go:356] "Generic (PLEG): container finished" podID="29011c56-f7dc-4418-8881-1cdfc9f277f1" containerID="3dc24560fd9c094f754551e2a4c39910244fe64bf041a874dcfd347d58f5b508" exitCode=0 Jun 10 10:44:53.046571 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:53.046080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerDied","Data":"3dc24560fd9c094f754551e2a4c39910244fe64bf041a874dcfd347d58f5b508"} Jun 10 10:44:53.047299 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:53.047272 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42gfm" event={"ID":"125476fa-5f69-45a5-8232-31541c6be892","Type":"ContainerStarted","Data":"ce628019c2442f98b5f1861ccba3d8184252f232e4cdc4393b61ec480af5fa1b"} Jun 10 10:44:53.444300 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:53.444258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:53.444484 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:53.444325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:53.444484 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:53.444422 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:44:53.444484 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:53.444431 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:44:53.444484 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:53.444454 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:44:53.444653 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:53.444492 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:55.444474717 +0000 UTC m=+38.093530669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:44:53.444653 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:53.444506 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:55.444500451 +0000 UTC m=+38.093556401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:44:54.052514 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:54.052480 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" event={"ID":"29011c56-f7dc-4418-8881-1cdfc9f277f1","Type":"ContainerStarted","Data":"5760487831bbfec22d87d22fbdb462f775b4fa0bcb89de6be4f965ec25f09966"} Jun 10 10:44:54.075080 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:54.075033 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mh7hw" podStartSLOduration=7.571016754 podStartE2EDuration="36.074998727s" totalImagesPullingTime="28.503981973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:54.07432344 +0000 UTC m=+36.723379413" watchObservedRunningTime="2026-06-10 10:44:54.074998727 +0000 UTC m=+36.724054699" Jun 10 10:44:54.229790 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:54.229756 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:54.326756 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:54.326725 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:54.865613 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:54.865430 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:54.918171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:54.918135 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:55.056150 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:55.056117 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42gfm" event={"ID":"125476fa-5f69-45a5-8232-31541c6be892","Type":"ContainerStarted","Data":"0e30937bd3a56deb3bcd8f06de52a816f6ba17f02143a57f2c330501d31014d4"} Jun 10 10:44:55.072322 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:55.072275 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-42gfm" podStartSLOduration=1.967925248 podStartE2EDuration="4.072258299s" totalImagesPullingTime="2.104333051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:44:55.071469705 +0000 UTC m=+37.720525677" watchObservedRunningTime="2026-06-10 10:44:55.072258299 +0000 UTC m=+37.721314271" Jun 10 10:44:55.462157 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:55.462116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:55.462356 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:55.462191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:55.462356 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:55.462282 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:44:55.462356 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:55.462301 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:44:55.462356 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:55.462283 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:44:55.462518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:55.462360 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:59.46234066 +0000 UTC m=+42.111396626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:44:55.462518 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:55.462373 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:44:59.462367277 +0000 UTC m=+42.111423228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:44:58.174383 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:58.174351 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:58.275483 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:58.275447 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:59.017234 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.017196 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:59.102589 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.102552 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:59.197629 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.197580 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:59.290112 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.290022 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:59.393141 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.393097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:44:59.492319 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.492285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:44:59.492466 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.492359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:44:59.492466 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:59.492446 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:44:59.492466 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:59.492457 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:44:59.492466 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:59.492458 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:44:59.492592 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:59.492515 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:07.492497335 +0000 UTC m=+50.141553301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:44:59.492592 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:44:59.492529 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:07.492523923 +0000 UTC m=+50.141579874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:44:59.957896 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:44:59.957851 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:45:00.008728 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:00.008690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:45:00.356669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:00.356638 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:45:00.425780 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:00.425742 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" limit=1024 Jun 10 10:45:07.554696 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:07.554654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:45:07.555242 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:07.554722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:45:07.555242 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:07.554842 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:07.555242 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:07.554840 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:45:07.555242 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:07.554863 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:45:07.555242 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:07.554897 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:23.554884344 +0000 UTC m=+66.203940295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:45:07.555242 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:07.554915 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:23.554902796 +0000 UTC m=+66.203958747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:45:20.042887 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:20.042858 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s2n5" Jun 10 10:45:23.578122 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.578076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:45:23.578122 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.578125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:45:23.578661 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.578172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:45:23.578661 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.578198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:45:23.578661 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:23.578222 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:45:23.578661 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:23.578242 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:45:23.578661 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:23.578303 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:55.57828742 +0000 UTC m=+98.227343371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:45:23.578661 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:23.578321 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:23.578661 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:23.578408 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:55.578392829 +0000 UTC m=+98.227448796 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:45:23.580692 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.580671 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 10 10:45:23.580753 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.580671 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 10 10:45:23.588470 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:23.588440 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 10 10:45:23.588593 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:23.588507 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:46:27.588490759 +0000 UTC m=+130.237546714 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : secret "metrics-daemon-secret" not found Jun 10 10:45:23.590332 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.590315 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 10 10:45:23.602671 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.602636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58t6\" (UniqueName: \"kubernetes.io/projected/bc6fff07-0a5d-4065-8912-789624912795-kube-api-access-q58t6\") pod \"network-check-target-ccsb6\" (UID: \"bc6fff07-0a5d-4065-8912-789624912795\") " pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:45:23.661291 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.661261 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8hbqv\"" Jun 10 10:45:23.670117 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.670081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:45:23.678520 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.678496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:45:23.681059 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.681041 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 10 10:45:23.691337 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.691309 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f2862d8-beef-4218-b972-896923c4709a-original-pull-secret\") pod \"global-pull-secret-syncer-ptw8k\" (UID: \"7f2862d8-beef-4218-b972-896923c4709a\") " pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:45:23.836691 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.836592 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccsb6"] Jun 10 10:45:23.841482 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:45:23.841455 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6fff07_0a5d_4065_8912_789624912795.slice/crio-063ea1295e5dec2972913096d5b1be4717bd812cdf651efdb1dbd82708de967f WatchSource:0}: Error finding container 063ea1295e5dec2972913096d5b1be4717bd812cdf651efdb1dbd82708de967f: Status 404 returned error can't find the container with id 063ea1295e5dec2972913096d5b1be4717bd812cdf651efdb1dbd82708de967f Jun 10 10:45:23.969372 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:23.969337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ptw8k" Jun 10 10:45:24.118149 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:24.118069 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ptw8k"] Jun 10 10:45:24.121516 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:45:24.121486 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2862d8_beef_4218_b972_896923c4709a.slice/crio-7fc067d10bd3a14cea694c3f982f4605a74045556bf08d325eedea5353dab5ab WatchSource:0}: Error finding container 7fc067d10bd3a14cea694c3f982f4605a74045556bf08d325eedea5353dab5ab: Status 404 returned error can't find the container with id 7fc067d10bd3a14cea694c3f982f4605a74045556bf08d325eedea5353dab5ab Jun 10 10:45:24.127621 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:24.127594 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ptw8k" event={"ID":"7f2862d8-beef-4218-b972-896923c4709a","Type":"ContainerStarted","Data":"7fc067d10bd3a14cea694c3f982f4605a74045556bf08d325eedea5353dab5ab"} Jun 10 10:45:24.128676 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:24.128655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccsb6" event={"ID":"bc6fff07-0a5d-4065-8912-789624912795","Type":"ContainerStarted","Data":"063ea1295e5dec2972913096d5b1be4717bd812cdf651efdb1dbd82708de967f"} Jun 10 10:45:29.143983 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:29.143945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ptw8k" event={"ID":"7f2862d8-beef-4218-b972-896923c4709a","Type":"ContainerStarted","Data":"f3252ebf1c9a19a40ee17371dcd1686f97df1887f5e1b41bf5ae2974ad97e457"} Jun 10 10:45:29.145351 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:29.145326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccsb6" event={"ID":"bc6fff07-0a5d-4065-8912-789624912795","Type":"ContainerStarted","Data":"6bbacc7cbd7f6dcb624ae7b79939da3a82d5409c31be3c43cb62dde4755257c5"} Jun 10 10:45:29.145454 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:29.145378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:45:29.157842 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:29.157680 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ptw8k" podStartSLOduration=67.075464322 podStartE2EDuration="1m11.157664206s" totalImagesPullingTime="4.082199884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:29.157156825 +0000 UTC m=+71.806213056" watchObservedRunningTime="2026-06-10 10:45:29.157664206 +0000 UTC m=+71.806720180" Jun 10 10:45:29.171180 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:29.171134 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ccsb6" podStartSLOduration=66.812213571 podStartE2EDuration="1m11.171121646s" totalImagesPullingTime="4.358908075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:29.170512044 +0000 UTC m=+71.819568017" watchObservedRunningTime="2026-06-10 10:45:29.171121646 +0000 UTC m=+71.820177621" Jun 10 10:45:55.609854 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:55.609784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:45:55.609854 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:45:55.609863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:45:55.610362 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:55.609944 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:45:55.610362 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:55.609965 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:45:55.610362 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:55.609968 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:55.610362 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:55.610021 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:46:59.610006518 +0000 UTC m=+162.259062468 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:45:55.610362 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:45:55.610046 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:46:59.610027779 +0000 UTC m=+162.259083736 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:46:00.149949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:46:00.149918 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ccsb6" Jun 10 10:46:27.643300 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:46:27.643245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:46:27.643837 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:27.643386 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 10 10:46:27.643837 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:27.643467 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:29.6434522 +0000 UTC m=+252.292508151 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : secret "metrics-daemon-secret" not found Jun 10 10:46:54.709200 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:54.709152 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" Jun 10 10:46:54.733483 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:54.733444 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vg7kl" podUID="cacd2e06-fcd6-43b0-aff9-6f114eddd501" Jun 10 10:46:54.865277 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:54.865228 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:46:55.357742 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:46:55.357704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vg7kl" Jun 10 10:46:55.357742 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:46:55.357747 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:46:59.659532 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:46:59.659484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:46:59.660054 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:46:59.659562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:46:59.660054 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:59.659650 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:46:59.660054 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:59.659714 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls podName:cacd2e06-fcd6-43b0-aff9-6f114eddd501 nodeName:}" failed. No retries permitted until 2026-06-10 10:49:01.659697228 +0000 UTC m=+284.308753182 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls") pod "dns-default-vg7kl" (UID: "cacd2e06-fcd6-43b0-aff9-6f114eddd501") : secret "dns-default-metrics-tls" not found Jun 10 10:46:59.660054 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:59.659654 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 10 10:46:59.660054 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:59.659752 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b6f49bc56-wpzlc: secret "image-registry-tls" not found Jun 10 10:46:59.660054 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:46:59.659822 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls podName:b3da4c40-3de7-4f9b-80c5-08d9042085e8 nodeName:}" failed. No retries permitted until 2026-06-10 10:49:01.659788335 +0000 UTC m=+284.308844303 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls") pod "image-registry-7b6f49bc56-wpzlc" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8") : secret "image-registry-tls" not found Jun 10 10:47:06.849546 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:47:06.849451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:48:12.972213 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.972175 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x"] Jun 10 10:48:12.975254 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.975227 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr"] Jun 10 10:48:12.975408 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.975371 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" Jun 10 10:48:12.977933 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.977915 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-896549976-rjfdv"] Jun 10 10:48:12.978056 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.978039 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:12.978549 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.978532 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zh95r\"" Jun 10 10:48:12.980153 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.980136 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jun 10 10:48:12.980739 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.980719 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb"] Jun 10 10:48:12.980886 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.980857 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jun 10 10:48:12.980958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.980908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:12.981013 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.980862 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9tpfd\"" Jun 10 10:48:12.981255 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.981233 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jun 10 10:48:12.983412 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.983397 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:12.988248 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.988229 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jun 10 10:48:12.988341 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.988273 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jun 10 10:48:12.988341 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.988284 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jun 10 10:48:12.988341 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.988295 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jun 10 10:48:12.988499 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.988352 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9928x\"" Jun 10 10:48:12.988499 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.988448 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jun 10 10:48:12.992107 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.989575 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jun 10 10:48:12.992107 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.990496 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-597zj\"" Jun 10 10:48:12.992107 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.991311 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jun 10 10:48:12.992107 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.991633 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jun 10 10:48:12.995750 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:12.995721 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jun 10 10:48:13.003228 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.003204 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x"] Jun 10 10:48:13.003989 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.003972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr"] Jun 10 10:48:13.005003 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.004986 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-896549976-rjfdv"] Jun 10 10:48:13.019122 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.019102 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb"] Jun 10 10:48:13.065861 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.065832 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc"] Jun 10 10:48:13.069147 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.069124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.070942 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.070919 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jun 10 10:48:13.071046 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.070985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mct8s\" (UniqueName: \"kubernetes.io/projected/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-kube-api-access-mct8s\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:13.071046 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfc1a540-76c8-4702-be98-ba0d4a1759f3-serving-cert\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.071143 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:13.071143 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f42b9d-d00f-4058-b109-072b242bc4e0-telemetry-config\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.071223 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsn56\" (UniqueName: \"kubernetes.io/projected/2e27c040-9441-4b68-b199-17cf79ee51d1-kube-api-access-tsn56\") pod \"network-check-source-dbd9cf844-9sl4x\" (UID: \"2e27c040-9441-4b68-b199-17cf79ee51d1\") " pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" Jun 10 10:48:13.071223 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc1a540-76c8-4702-be98-ba0d4a1759f3-tmp\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.071297 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqlh4\" (UniqueName: \"kubernetes.io/projected/dfc1a540-76c8-4702-be98-ba0d4a1759f3-kube-api-access-hqlh4\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.071297 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.071297 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc1a540-76c8-4702-be98-ba0d4a1759f3-trusted-ca-bundle\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.071438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4qb\" (UniqueName: \"kubernetes.io/projected/20f42b9d-d00f-4058-b109-072b242bc4e0-kube-api-access-bn4qb\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.071438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dfc1a540-76c8-4702-be98-ba0d4a1759f3-snapshots\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.071438 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc1a540-76c8-4702-be98-ba0d4a1759f3-service-ca-bundle\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.071645 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071630 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jun 10 10:48:13.071704 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071657 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jun 10 10:48:13.071704 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071675 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jun 10 10:48:13.071856 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.071722 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nf5d6\"" Jun 10 10:48:13.087531 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.087508 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc"] Jun 10 10:48:13.159678 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.159645 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-668fdbf85c-9q6dc"] Jun 10 10:48:13.162470 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.162452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.165636 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.165615 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jun 10 10:48:13.165769 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.165732 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jun 10 10:48:13.165769 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.165764 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jun 10 10:48:13.165908 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.165844 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jun 10 10:48:13.167648 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.167626 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7btx5\"" Jun 10 10:48:13.171345 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171325 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jun 10 10:48:13.171635 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mct8s\" (UniqueName: \"kubernetes.io/projected/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-kube-api-access-mct8s\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:13.171708 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfc1a540-76c8-4702-be98-ba0d4a1759f3-serving-cert\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.171708 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f42b9d-d00f-4058-b109-072b242bc4e0-telemetry-config\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.171708 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4075f9cd-34d0-4601-b9ed-227e3918826e-config\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.171889 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqlh4\" (UniqueName: \"kubernetes.io/projected/dfc1a540-76c8-4702-be98-ba0d4a1759f3-kube-api-access-hqlh4\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.171952 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4075f9cd-34d0-4601-b9ed-227e3918826e-serving-cert\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.171995 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.171968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc1a540-76c8-4702-be98-ba0d4a1759f3-tmp\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.172078 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:13.172172 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsn56\" (UniqueName: \"kubernetes.io/projected/2e27c040-9441-4b68-b199-17cf79ee51d1-kube-api-access-tsn56\") pod \"network-check-source-dbd9cf844-9sl4x\" (UID: \"2e27c040-9441-4b68-b199-17cf79ee51d1\") " pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" Jun 10 10:48:13.172172 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.172121 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 10 10:48:13.172172 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.172172 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc1a540-76c8-4702-be98-ba0d4a1759f3-trusted-ca-bundle\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.172378 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.172187 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls podName:34ea66eb-4140-4b6c-9071-54a31a0e2cf4 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:13.672166942 +0000 UTC m=+236.321222893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-4snxr" (UID: "34ea66eb-4140-4b6c-9071-54a31a0e2cf4") : secret "samples-operator-tls" not found Jun 10 10:48:13.172378 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.172248 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:13.172378 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.172300 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls podName:20f42b9d-d00f-4058-b109-072b242bc4e0 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:13.672283666 +0000 UTC m=+236.321339624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-74bbf69bbb-4gmkb" (UID: "20f42b9d-d00f-4058-b109-072b242bc4e0") : secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:13.172378 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc1a540-76c8-4702-be98-ba0d4a1759f3-tmp\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.172378 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4qb\" (UniqueName: \"kubernetes.io/projected/20f42b9d-d00f-4058-b109-072b242bc4e0-kube-api-access-bn4qb\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.172635 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dfc1a540-76c8-4702-be98-ba0d4a1759f3-snapshots\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.172635 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc1a540-76c8-4702-be98-ba0d4a1759f3-service-ca-bundle\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.172635 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/20f42b9d-d00f-4058-b109-072b242bc4e0-telemetry-config\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.172635 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvwj\" (UniqueName: \"kubernetes.io/projected/4075f9cd-34d0-4601-b9ed-227e3918826e-kube-api-access-qhvwj\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.172899 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dfc1a540-76c8-4702-be98-ba0d4a1759f3-snapshots\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.172954 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.172883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc1a540-76c8-4702-be98-ba0d4a1759f3-service-ca-bundle\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.173247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.173225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfc1a540-76c8-4702-be98-ba0d4a1759f3-trusted-ca-bundle\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.174239 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.174216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfc1a540-76c8-4702-be98-ba0d4a1759f3-serving-cert\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.174478 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.174462 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-668fdbf85c-9q6dc"] Jun 10 10:48:13.181168 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.181140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqlh4\" (UniqueName: \"kubernetes.io/projected/dfc1a540-76c8-4702-be98-ba0d4a1759f3-kube-api-access-hqlh4\") pod \"insights-operator-896549976-rjfdv\" (UID: \"dfc1a540-76c8-4702-be98-ba0d4a1759f3\") " pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.182023 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.182005 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsn56\" (UniqueName: \"kubernetes.io/projected/2e27c040-9441-4b68-b199-17cf79ee51d1-kube-api-access-tsn56\") pod \"network-check-source-dbd9cf844-9sl4x\" (UID: \"2e27c040-9441-4b68-b199-17cf79ee51d1\") " pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" Jun 10 10:48:13.182142 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.182124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mct8s\" (UniqueName: \"kubernetes.io/projected/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-kube-api-access-mct8s\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:13.182208 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.182150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4qb\" (UniqueName: \"kubernetes.io/projected/20f42b9d-d00f-4058-b109-072b242bc4e0-kube-api-access-bn4qb\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.273628 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.273537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4513f-e58c-41c5-bda8-f64927fd7358-trusted-ca\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.273628 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.273577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4075f9cd-34d0-4601-b9ed-227e3918826e-serving-cert\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.273628 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.273598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7c4513f-e58c-41c5-bda8-f64927fd7358-config\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.273926 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.273634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7c4513f-e58c-41c5-bda8-f64927fd7358-serving-cert\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.273926 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.273664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrq6\" (UniqueName: \"kubernetes.io/projected/d7c4513f-e58c-41c5-bda8-f64927fd7358-kube-api-access-cjrq6\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.273926 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.273689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvwj\" (UniqueName: \"kubernetes.io/projected/4075f9cd-34d0-4601-b9ed-227e3918826e-kube-api-access-qhvwj\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.273926 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.273770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4075f9cd-34d0-4601-b9ed-227e3918826e-config\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.274266 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.274250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4075f9cd-34d0-4601-b9ed-227e3918826e-config\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.275935 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.275919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4075f9cd-34d0-4601-b9ed-227e3918826e-serving-cert\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.281963 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.281939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvwj\" (UniqueName: \"kubernetes.io/projected/4075f9cd-34d0-4601-b9ed-227e3918826e-kube-api-access-qhvwj\") pod \"kube-storage-version-migrator-operator-55bb7c4d49-pszhc\" (UID: \"4075f9cd-34d0-4601-b9ed-227e3918826e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.284864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.284846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" Jun 10 10:48:13.298717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.298698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-896549976-rjfdv" Jun 10 10:48:13.375663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.374448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4513f-e58c-41c5-bda8-f64927fd7358-trusted-ca\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.375663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.374497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7c4513f-e58c-41c5-bda8-f64927fd7358-config\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.375663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.374550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7c4513f-e58c-41c5-bda8-f64927fd7358-serving-cert\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.375663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.374603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrq6\" (UniqueName: \"kubernetes.io/projected/d7c4513f-e58c-41c5-bda8-f64927fd7358-kube-api-access-cjrq6\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.375663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.375444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4513f-e58c-41c5-bda8-f64927fd7358-trusted-ca\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.375663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.375619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7c4513f-e58c-41c5-bda8-f64927fd7358-config\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.378051 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.378033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" Jun 10 10:48:13.378473 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.378441 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7c4513f-e58c-41c5-bda8-f64927fd7358-serving-cert\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.384350 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.384232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrq6\" (UniqueName: \"kubernetes.io/projected/d7c4513f-e58c-41c5-bda8-f64927fd7358-kube-api-access-cjrq6\") pod \"console-operator-668fdbf85c-9q6dc\" (UID: \"d7c4513f-e58c-41c5-bda8-f64927fd7358\") " pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.417529 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.417495 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x"] Jun 10 10:48:13.420912 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:13.420880 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e27c040_9441_4b68_b199_17cf79ee51d1.slice/crio-b4ff7607f8cfe02a896e2826d8b12c2901158feefbc4ce20f1177df5cea9a2dc WatchSource:0}: Error finding container b4ff7607f8cfe02a896e2826d8b12c2901158feefbc4ce20f1177df5cea9a2dc: Status 404 returned error can't find the container with id b4ff7607f8cfe02a896e2826d8b12c2901158feefbc4ce20f1177df5cea9a2dc Jun 10 10:48:13.435371 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.435343 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-896549976-rjfdv"] Jun 10 10:48:13.447345 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:13.447307 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc1a540_76c8_4702_be98_ba0d4a1759f3.slice/crio-ebffb6aedeaadc0c53bd6bff44afef1fd432c5d96a1ec901f24beb8c2ad99f7f WatchSource:0}: Error finding container ebffb6aedeaadc0c53bd6bff44afef1fd432c5d96a1ec901f24beb8c2ad99f7f: Status 404 returned error can't find the container with id ebffb6aedeaadc0c53bd6bff44afef1fd432c5d96a1ec901f24beb8c2ad99f7f Jun 10 10:48:13.471299 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.471065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:13.523389 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.523304 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc"] Jun 10 10:48:13.529560 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:13.529508 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4075f9cd_34d0_4601_b9ed_227e3918826e.slice/crio-6c471b15db0c8c1d1f65e14f3ca84b7aaf95dd2f2d77743e6aab9ef2a0c13b43 WatchSource:0}: Error finding container 6c471b15db0c8c1d1f65e14f3ca84b7aaf95dd2f2d77743e6aab9ef2a0c13b43: Status 404 returned error can't find the container with id 6c471b15db0c8c1d1f65e14f3ca84b7aaf95dd2f2d77743e6aab9ef2a0c13b43 Jun 10 10:48:13.551247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.551164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-rjfdv" event={"ID":"dfc1a540-76c8-4702-be98-ba0d4a1759f3","Type":"ContainerStarted","Data":"ebffb6aedeaadc0c53bd6bff44afef1fd432c5d96a1ec901f24beb8c2ad99f7f"} Jun 10 10:48:13.552868 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.552838 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" event={"ID":"2e27c040-9441-4b68-b199-17cf79ee51d1","Type":"ContainerStarted","Data":"19ca30f84c5514e1e5a7d1e2a21e638c4ee8174a7cfc62449729e421f3c26dca"} Jun 10 10:48:13.553034 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.552887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" event={"ID":"2e27c040-9441-4b68-b199-17cf79ee51d1","Type":"ContainerStarted","Data":"b4ff7607f8cfe02a896e2826d8b12c2901158feefbc4ce20f1177df5cea9a2dc"} Jun 10 10:48:13.554135 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.554111 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" event={"ID":"4075f9cd-34d0-4601-b9ed-227e3918826e","Type":"ContainerStarted","Data":"6c471b15db0c8c1d1f65e14f3ca84b7aaf95dd2f2d77743e6aab9ef2a0c13b43"} Jun 10 10:48:13.571653 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.571591 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-dbd9cf844-9sl4x" podStartSLOduration=1.571570286 podStartE2EDuration="1.571570286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:13.570995526 +0000 UTC m=+236.220051500" watchObservedRunningTime="2026-06-10 10:48:13.571570286 +0000 UTC m=+236.220626259" Jun 10 10:48:13.629275 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.629240 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-668fdbf85c-9q6dc"] Jun 10 10:48:13.632440 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:13.632413 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c4513f_e58c_41c5_bda8_f64927fd7358.slice/crio-0f95e90accf23d6ca8dec15fc5e265142ebcd44a7ce5681d4f8757ac7e5c2690 WatchSource:0}: Error finding container 0f95e90accf23d6ca8dec15fc5e265142ebcd44a7ce5681d4f8757ac7e5c2690: Status 404 returned error can't find the container with id 0f95e90accf23d6ca8dec15fc5e265142ebcd44a7ce5681d4f8757ac7e5c2690 Jun 10 10:48:13.677199 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.677162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:13.677394 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:13.677220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:13.677394 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.677334 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:13.677394 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.677334 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 10 10:48:13.677513 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.677399 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls podName:34ea66eb-4140-4b6c-9071-54a31a0e2cf4 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:14.677383356 +0000 UTC m=+237.326439327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-4snxr" (UID: "34ea66eb-4140-4b6c-9071-54a31a0e2cf4") : secret "samples-operator-tls" not found Jun 10 10:48:13.677513 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:13.677412 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls podName:20f42b9d-d00f-4058-b109-072b242bc4e0 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:14.677407019 +0000 UTC m=+237.326462970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-74bbf69bbb-4gmkb" (UID: "20f42b9d-d00f-4058-b109-072b242bc4e0") : secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:14.559754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:14.559690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" event={"ID":"d7c4513f-e58c-41c5-bda8-f64927fd7358","Type":"ContainerStarted","Data":"0f95e90accf23d6ca8dec15fc5e265142ebcd44a7ce5681d4f8757ac7e5c2690"} Jun 10 10:48:14.691101 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:14.691061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:14.691311 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:14.691135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:14.691311 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:14.691239 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 10 10:48:14.691311 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:14.691283 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:14.691475 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:14.691329 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls podName:34ea66eb-4140-4b6c-9071-54a31a0e2cf4 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:16.691307241 +0000 UTC m=+239.340363203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-4snxr" (UID: "34ea66eb-4140-4b6c-9071-54a31a0e2cf4") : secret "samples-operator-tls" not found Jun 10 10:48:14.691475 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:14.691349 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls podName:20f42b9d-d00f-4058-b109-072b242bc4e0 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:16.691339626 +0000 UTC m=+239.340395581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-74bbf69bbb-4gmkb" (UID: "20f42b9d-d00f-4058-b109-072b242bc4e0") : secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:16.566142 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.566104 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" event={"ID":"4075f9cd-34d0-4601-b9ed-227e3918826e","Type":"ContainerStarted","Data":"0f92d7d2e0c4796013e1e88b31ea51139f12a0250bedabe552e0ab8e755a344f"} Jun 10 10:48:16.567719 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.567697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" event={"ID":"d7c4513f-e58c-41c5-bda8-f64927fd7358","Type":"ContainerStarted","Data":"c98d4071aa47911f82afeeb5740d8b93b248e1ceca377aa8739e16efbbc54247"} Jun 10 10:48:16.567879 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.567758 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:16.569083 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.569058 2578 patch_prober.go:28] interesting pod/console-operator-668fdbf85c-9q6dc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Jun 10 10:48:16.569237 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.569102 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" podUID="d7c4513f-e58c-41c5-bda8-f64927fd7358" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" Jun 10 10:48:16.569946 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.569917 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-rjfdv" event={"ID":"dfc1a540-76c8-4702-be98-ba0d4a1759f3","Type":"ContainerStarted","Data":"b900d3671734e5f0b89709961cdcf2b85cc039d88690b184a246e91338794fc6"} Jun 10 10:48:16.583019 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.582948 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" podStartSLOduration=0.628866499 podStartE2EDuration="3.582930436s" totalImagesPullingTime="2.954063937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:16.582010263 +0000 UTC m=+239.231066232" watchObservedRunningTime="2026-06-10 10:48:16.582930436 +0000 UTC m=+239.231986441" Jun 10 10:48:16.598868 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.598758 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-896549976-rjfdv" podStartSLOduration=1.5680604169999999 podStartE2EDuration="4.598739715s" totalImagesPullingTime="3.030679298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:16.59763122 +0000 UTC m=+239.246687194" watchObservedRunningTime="2026-06-10 10:48:16.598739715 +0000 UTC m=+239.247795687" Jun 10 10:48:16.613691 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.613606 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" podStartSLOduration=0.765332703 podStartE2EDuration="3.613577795s" totalImagesPullingTime="2.848245092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:16.613056784 +0000 UTC m=+239.262112760" watchObservedRunningTime="2026-06-10 10:48:16.613577795 +0000 UTC m=+239.262633768" Jun 10 10:48:16.711810 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.711166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:16.711810 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:16.711256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:16.711810 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:16.711545 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:16.711810 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:16.711634 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls podName:20f42b9d-d00f-4058-b109-072b242bc4e0 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:20.711613342 +0000 UTC m=+243.360669296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-74bbf69bbb-4gmkb" (UID: "20f42b9d-d00f-4058-b109-072b242bc4e0") : secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:16.711810 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:16.711670 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 10 10:48:16.712199 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:16.711893 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls podName:34ea66eb-4140-4b6c-9071-54a31a0e2cf4 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:20.711875546 +0000 UTC m=+243.360931498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-4snxr" (UID: "34ea66eb-4140-4b6c-9071-54a31a0e2cf4") : secret "samples-operator-tls" not found Jun 10 10:48:17.575382 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:17.575356 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/0.log" Jun 10 10:48:17.575773 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:17.575395 2578 generic.go:356] "Generic (PLEG): container finished" podID="d7c4513f-e58c-41c5-bda8-f64927fd7358" containerID="c98d4071aa47911f82afeeb5740d8b93b248e1ceca377aa8739e16efbbc54247" exitCode=255 Jun 10 10:48:17.575773 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:17.575434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" event={"ID":"d7c4513f-e58c-41c5-bda8-f64927fd7358","Type":"ContainerDied","Data":"c98d4071aa47911f82afeeb5740d8b93b248e1ceca377aa8739e16efbbc54247"} Jun 10 10:48:17.576004 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:17.575772 2578 scope.go:117] "RemoveContainer" containerID="c98d4071aa47911f82afeeb5740d8b93b248e1ceca377aa8739e16efbbc54247" Jun 10 10:48:18.064094 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.064068 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc"] Jun 10 10:48:18.067072 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.067056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" Jun 10 10:48:18.069017 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.068993 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jun 10 10:48:18.069140 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.069039 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jun 10 10:48:18.069140 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.069045 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wsksq\"" Jun 10 10:48:18.076450 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.076085 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc"] Jun 10 10:48:18.223521 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.223485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8j64\" (UniqueName: \"kubernetes.io/projected/53b6242f-c082-471c-8238-e57e35a3d780-kube-api-access-m8j64\") pod \"migrator-955ffdf5f-8d2zc\" (UID: \"53b6242f-c082-471c-8238-e57e35a3d780\") " pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" Jun 10 10:48:18.324686 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.324612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8j64\" (UniqueName: \"kubernetes.io/projected/53b6242f-c082-471c-8238-e57e35a3d780-kube-api-access-m8j64\") pod \"migrator-955ffdf5f-8d2zc\" (UID: \"53b6242f-c082-471c-8238-e57e35a3d780\") " pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" Jun 10 10:48:18.333416 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.333389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8j64\" (UniqueName: \"kubernetes.io/projected/53b6242f-c082-471c-8238-e57e35a3d780-kube-api-access-m8j64\") pod \"migrator-955ffdf5f-8d2zc\" (UID: \"53b6242f-c082-471c-8238-e57e35a3d780\") " pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" Jun 10 10:48:18.377283 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.377245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" Jun 10 10:48:18.495719 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.495687 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc"] Jun 10 10:48:18.499901 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:18.499867 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b6242f_c082_471c_8238_e57e35a3d780.slice/crio-8fb7182b2456423ca3194dd0894fe511e5cec995ce4c6e496b9e60cf84ed579a WatchSource:0}: Error finding container 8fb7182b2456423ca3194dd0894fe511e5cec995ce4c6e496b9e60cf84ed579a: Status 404 returned error can't find the container with id 8fb7182b2456423ca3194dd0894fe511e5cec995ce4c6e496b9e60cf84ed579a Jun 10 10:48:18.579532 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.579471 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:48:18.579932 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.579913 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/0.log" Jun 10 10:48:18.579995 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.579959 2578 generic.go:356] "Generic (PLEG): container finished" podID="d7c4513f-e58c-41c5-bda8-f64927fd7358" containerID="2166a87d76b1c22a489e133c1666ad53e4a4cc28ff1c630a5540c733c9eecbb8" exitCode=255 Jun 10 10:48:18.580060 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.580038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" event={"ID":"d7c4513f-e58c-41c5-bda8-f64927fd7358","Type":"ContainerDied","Data":"2166a87d76b1c22a489e133c1666ad53e4a4cc28ff1c630a5540c733c9eecbb8"} Jun 10 10:48:18.580101 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.580095 2578 scope.go:117] "RemoveContainer" containerID="c98d4071aa47911f82afeeb5740d8b93b248e1ceca377aa8739e16efbbc54247" Jun 10 10:48:18.580361 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.580335 2578 scope.go:117] "RemoveContainer" containerID="2166a87d76b1c22a489e133c1666ad53e4a4cc28ff1c630a5540c733c9eecbb8" Jun 10 10:48:18.580556 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:18.580538 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-668fdbf85c-9q6dc_openshift-console-operator(d7c4513f-e58c-41c5-bda8-f64927fd7358)\"" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" podUID="d7c4513f-e58c-41c5-bda8-f64927fd7358" Jun 10 10:48:18.581056 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:18.581033 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" event={"ID":"53b6242f-c082-471c-8238-e57e35a3d780","Type":"ContainerStarted","Data":"8fb7182b2456423ca3194dd0894fe511e5cec995ce4c6e496b9e60cf84ed579a"} Jun 10 10:48:19.511914 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.511879 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7d74d578b8-j8dc9"] Jun 10 10:48:19.514673 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.514657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.516873 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.516851 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jun 10 10:48:19.517330 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.517300 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jun 10 10:48:19.517433 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.517398 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jun 10 10:48:19.517433 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.517413 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jun 10 10:48:19.517548 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.517398 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nj4x9\"" Jun 10 10:48:19.525041 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.525021 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7d74d578b8-j8dc9"] Jun 10 10:48:19.585690 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.585671 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:48:19.586102 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.586001 2578 scope.go:117] "RemoveContainer" containerID="2166a87d76b1c22a489e133c1666ad53e4a4cc28ff1c630a5540c733c9eecbb8" Jun 10 10:48:19.586175 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:19.586146 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-668fdbf85c-9q6dc_openshift-console-operator(d7c4513f-e58c-41c5-bda8-f64927fd7358)\"" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" podUID="d7c4513f-e58c-41c5-bda8-f64927fd7358" Jun 10 10:48:19.634624 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.634600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-signing-cabundle\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.634776 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.634717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-signing-key\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.634887 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.634826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mqn\" (UniqueName: \"kubernetes.io/projected/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-kube-api-access-m5mqn\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.735680 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.735644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mqn\" (UniqueName: \"kubernetes.io/projected/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-kube-api-access-m5mqn\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.735864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.735719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-signing-cabundle\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.735864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.735783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-signing-key\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.736353 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.736334 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-signing-cabundle\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.738271 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.738257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-signing-key\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.743304 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.743283 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mqn\" (UniqueName: \"kubernetes.io/projected/d4c45da6-29dc-4e76-a256-1ae9fa85f48b-kube-api-access-m5mqn\") pod \"service-ca-7d74d578b8-j8dc9\" (UID: \"d4c45da6-29dc-4e76-a256-1ae9fa85f48b\") " pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.823577 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.823541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" Jun 10 10:48:19.950651 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:19.950621 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7d74d578b8-j8dc9"] Jun 10 10:48:19.953699 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:19.953674 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c45da6_29dc_4e76_a256_1ae9fa85f48b.slice/crio-089f8d00c14badfbacfef9e42a0dc800b9e2de0019816075a29aa585cf56113a WatchSource:0}: Error finding container 089f8d00c14badfbacfef9e42a0dc800b9e2de0019816075a29aa585cf56113a: Status 404 returned error can't find the container with id 089f8d00c14badfbacfef9e42a0dc800b9e2de0019816075a29aa585cf56113a Jun 10 10:48:20.592666 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:20.592621 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" event={"ID":"53b6242f-c082-471c-8238-e57e35a3d780","Type":"ContainerStarted","Data":"14111a0d421c79e5b4bffe5d2c0396992a8c08e021a07d98dd930c524a875cec"} Jun 10 10:48:20.593106 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:20.592692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" event={"ID":"53b6242f-c082-471c-8238-e57e35a3d780","Type":"ContainerStarted","Data":"7f18926f20793a06440f8fb116edd8a03eb9e54fc4d27e4699b4cc42f9042b95"} Jun 10 10:48:20.593916 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:20.593894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" event={"ID":"d4c45da6-29dc-4e76-a256-1ae9fa85f48b","Type":"ContainerStarted","Data":"089f8d00c14badfbacfef9e42a0dc800b9e2de0019816075a29aa585cf56113a"} Jun 10 10:48:20.619454 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:20.619391 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-8d2zc" podStartSLOduration=1.5768156960000002 podStartE2EDuration="2.619368121s" totalImagesPullingTime="1.042552425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:20.617936399 +0000 UTC m=+243.266992372" watchObservedRunningTime="2026-06-10 10:48:20.619368121 +0000 UTC m=+243.268424096" Jun 10 10:48:20.714620 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:20.714589 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-29rcg_6bffc05c-b337-483c-8b1f-907780f2d0d2/dns-node-resolver/0.log" Jun 10 10:48:20.743736 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:20.743695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:20.743925 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:20.743753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:20.743925 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:20.743864 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 10 10:48:20.743925 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:20.743907 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:20.743925 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:20.743926 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls podName:34ea66eb-4140-4b6c-9071-54a31a0e2cf4 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:28.743909462 +0000 UTC m=+251.392965413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-4snxr" (UID: "34ea66eb-4140-4b6c-9071-54a31a0e2cf4") : secret "samples-operator-tls" not found Jun 10 10:48:20.744110 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:20.743958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls podName:20f42b9d-d00f-4058-b109-072b242bc4e0 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:28.743942542 +0000 UTC m=+251.392998504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-74bbf69bbb-4gmkb" (UID: "20f42b9d-d00f-4058-b109-072b242bc4e0") : secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:22.115162 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:22.115138 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ps8th_12b531fb-fb79-47d1-adab-9eca526c52de/node-ca/0.log" Jun 10 10:48:22.518728 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:22.518646 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42gfm_125476fa-5f69-45a5-8232-31541c6be892/serve-healthcheck-canary/0.log" Jun 10 10:48:22.601713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:22.601677 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" event={"ID":"d4c45da6-29dc-4e76-a256-1ae9fa85f48b","Type":"ContainerStarted","Data":"d5dee9cc2db2c3d7de214262f0cff2f4164a0b42b961f78b1948aae5a22f5410"} Jun 10 10:48:23.115894 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:23.115856 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-955ffdf5f-8d2zc_53b6242f-c082-471c-8238-e57e35a3d780/migrator/0.log" Jun 10 10:48:23.320779 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:23.320753 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-955ffdf5f-8d2zc_53b6242f-c082-471c-8238-e57e35a3d780/graceful-termination/0.log" Jun 10 10:48:23.471742 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:23.471701 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:23.472139 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:23.472122 2578 scope.go:117] "RemoveContainer" containerID="2166a87d76b1c22a489e133c1666ad53e4a4cc28ff1c630a5540c733c9eecbb8" Jun 10 10:48:23.472309 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:23.472288 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-668fdbf85c-9q6dc_openshift-console-operator(d7c4513f-e58c-41c5-bda8-f64927fd7358)\"" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" podUID="d7c4513f-e58c-41c5-bda8-f64927fd7358" Jun 10 10:48:23.516822 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:23.516770 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55bb7c4d49-pszhc_4075f9cd-34d0-4601-b9ed-227e3918826e/kube-storage-version-migrator-operator/0.log" Jun 10 10:48:26.568071 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:26.568029 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:26.568453 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:26.568436 2578 scope.go:117] "RemoveContainer" containerID="2166a87d76b1c22a489e133c1666ad53e4a4cc28ff1c630a5540c733c9eecbb8" Jun 10 10:48:26.568599 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:26.568579 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-668fdbf85c-9q6dc_openshift-console-operator(d7c4513f-e58c-41c5-bda8-f64927fd7358)\"" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" podUID="d7c4513f-e58c-41c5-bda8-f64927fd7358" Jun 10 10:48:28.806723 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:28.806688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:28.807102 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:28.806730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:28.807102 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:28.806858 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:28.807102 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:28.806914 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls podName:20f42b9d-d00f-4058-b109-072b242bc4e0 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:44.80689926 +0000 UTC m=+267.455955210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-74bbf69bbb-4gmkb" (UID: "20f42b9d-d00f-4058-b109-072b242bc4e0") : secret "cluster-monitoring-operator-tls" not found Jun 10 10:48:28.809199 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:28.809176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/34ea66eb-4140-4b6c-9071-54a31a0e2cf4-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-4snxr\" (UID: \"34ea66eb-4140-4b6c-9071-54a31a0e2cf4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:28.893412 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:28.893382 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9tpfd\"" Jun 10 10:48:28.902015 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:28.901993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" Jun 10 10:48:29.030048 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:29.029587 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7d74d578b8-j8dc9" podStartSLOduration=8.430845235 podStartE2EDuration="10.029563987s" totalImagesPullingTime="1.598718752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:22.639992897 +0000 UTC m=+245.289048871" watchObservedRunningTime="2026-06-10 10:48:29.029563987 +0000 UTC m=+251.678619961" Jun 10 10:48:29.030048 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:29.029976 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr"] Jun 10 10:48:29.625868 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:29.625832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" event={"ID":"34ea66eb-4140-4b6c-9071-54a31a0e2cf4","Type":"ContainerStarted","Data":"5328f0ee663a18aabf02760cf2d4fe0c243dafe46e2c4c09cd2d65d1a1df9ed8"} Jun 10 10:48:29.712610 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:29.712562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:48:29.712816 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:29.712689 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 10 10:48:29.712816 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:29.712764 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs podName:8327523a-eb26-4c9d-847a-761b868b2d27 nodeName:}" failed. No retries permitted until 2026-06-10 10:50:31.712744806 +0000 UTC m=+374.361800769 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs") pod "network-metrics-daemon-8t72g" (UID: "8327523a-eb26-4c9d-847a-761b868b2d27") : secret "metrics-daemon-secret" not found Jun 10 10:48:31.639327 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:31.639295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" event={"ID":"34ea66eb-4140-4b6c-9071-54a31a0e2cf4","Type":"ContainerStarted","Data":"3c38df9300c322aba8032552d64b5b82fa21344917a3d70020229410d8469773"} Jun 10 10:48:31.639707 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:31.639373 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" event={"ID":"34ea66eb-4140-4b6c-9071-54a31a0e2cf4","Type":"ContainerStarted","Data":"eb126d506d3b3a5b6af631fcaa2f3f85dcca6f3dd28c707addce684dad4af05d"} Jun 10 10:48:31.655934 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:31.655882 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-4snxr" podStartSLOduration=18.043309905 podStartE2EDuration="19.65586107s" totalImagesPullingTime="1.612551165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:31.654092488 +0000 UTC m=+254.303148462" watchObservedRunningTime="2026-06-10 10:48:31.65586107 +0000 UTC m=+254.304917044" Jun 10 10:48:38.919649 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.919569 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j7c8x"] Jun 10 10:48:38.923151 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.923133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:38.927275 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.927254 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jun 10 10:48:38.927563 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.927546 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4h7qz\"" Jun 10 10:48:38.927618 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.927547 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jun 10 10:48:38.942281 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.942249 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7c8x"] Jun 10 10:48:38.988316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.988283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xcb\" (UniqueName: \"kubernetes.io/projected/88b447cf-9b8e-4774-b015-f23fd9426eba-kube-api-access-t8xcb\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:38.988482 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.988333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b447cf-9b8e-4774-b015-f23fd9426eba-crio-socket\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:38.988482 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.988423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b447cf-9b8e-4774-b015-f23fd9426eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:38.988482 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.988465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b447cf-9b8e-4774-b015-f23fd9426eba-data-volume\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:38.988585 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:38.988501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b447cf-9b8e-4774-b015-f23fd9426eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.089469 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.089432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b447cf-9b8e-4774-b015-f23fd9426eba-data-volume\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.089632 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.089476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b447cf-9b8e-4774-b015-f23fd9426eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.089678 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.089632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xcb\" (UniqueName: \"kubernetes.io/projected/88b447cf-9b8e-4774-b015-f23fd9426eba-kube-api-access-t8xcb\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.089678 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.089673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b447cf-9b8e-4774-b015-f23fd9426eba-crio-socket\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.089764 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.089737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b447cf-9b8e-4774-b015-f23fd9426eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.089863 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.089844 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b447cf-9b8e-4774-b015-f23fd9426eba-data-volume\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.089932 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.089867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b447cf-9b8e-4774-b015-f23fd9426eba-crio-socket\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.090077 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.090060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b447cf-9b8e-4774-b015-f23fd9426eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.092167 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.092150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b447cf-9b8e-4774-b015-f23fd9426eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.100269 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.100248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xcb\" (UniqueName: \"kubernetes.io/projected/88b447cf-9b8e-4774-b015-f23fd9426eba-kube-api-access-t8xcb\") pod \"insights-runtime-extractor-j7c8x\" (UID: \"88b447cf-9b8e-4774-b015-f23fd9426eba\") " pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.232946 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.232856 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7c8x" Jun 10 10:48:39.363583 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.363548 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7c8x"] Jun 10 10:48:39.366672 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:39.366643 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b447cf_9b8e_4774_b015_f23fd9426eba.slice/crio-e13e1ca84c058d3f03fe2a46a83315637ce73e97fe4d599941fe79beaed7c8fe WatchSource:0}: Error finding container e13e1ca84c058d3f03fe2a46a83315637ce73e97fe4d599941fe79beaed7c8fe: Status 404 returned error can't find the container with id e13e1ca84c058d3f03fe2a46a83315637ce73e97fe4d599941fe79beaed7c8fe Jun 10 10:48:39.663813 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.663773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7c8x" event={"ID":"88b447cf-9b8e-4774-b015-f23fd9426eba","Type":"ContainerStarted","Data":"d1e8c5d459b351afd14e9c8c04dadf38b79ba85783f0a2e33600cfaba57a19e4"} Jun 10 10:48:39.663994 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.663850 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7c8x" event={"ID":"88b447cf-9b8e-4774-b015-f23fd9426eba","Type":"ContainerStarted","Data":"e13e1ca84c058d3f03fe2a46a83315637ce73e97fe4d599941fe79beaed7c8fe"} Jun 10 10:48:39.850061 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:39.850029 2578 scope.go:117] "RemoveContainer" containerID="2166a87d76b1c22a489e133c1666ad53e4a4cc28ff1c630a5540c733c9eecbb8" Jun 10 10:48:40.674358 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:40.674178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7c8x" event={"ID":"88b447cf-9b8e-4774-b015-f23fd9426eba","Type":"ContainerStarted","Data":"104f21c368bed4eb384dfadd6945c8b587d2456f0856c28f6c12ba0db846a8ee"} Jun 10 10:48:40.676270 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:40.676241 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:48:40.676400 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:40.676323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" event={"ID":"d7c4513f-e58c-41c5-bda8-f64927fd7358","Type":"ContainerStarted","Data":"434a915ae22ddebdd623e8dc83cae2ba16ff6aa24a27ad6624f32f6b98379096"} Jun 10 10:48:40.676614 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:40.676593 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:40.682917 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:40.682891 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-668fdbf85c-9q6dc" Jun 10 10:48:42.690886 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:42.690846 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7c8x" event={"ID":"88b447cf-9b8e-4774-b015-f23fd9426eba","Type":"ContainerStarted","Data":"d80b9623148a6a85db099afdf6b0711a9603a51297ddcc38c3908ca15ebcde40"} Jun 10 10:48:42.712128 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:42.712086 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j7c8x" podStartSLOduration=1.785503513 podStartE2EDuration="4.712074131s" totalImagesPullingTime="2.926570618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:48:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:42.71127243 +0000 UTC m=+265.360328403" watchObservedRunningTime="2026-06-10 10:48:42.712074131 +0000 UTC m=+265.361130104" Jun 10 10:48:44.832839 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:44.832768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:44.835236 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:44.835212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/20f42b9d-d00f-4058-b109-072b242bc4e0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-74bbf69bbb-4gmkb\" (UID: \"20f42b9d-d00f-4058-b109-072b242bc4e0\") " pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:45.106538 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:45.106460 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-597zj\"" Jun 10 10:48:45.114897 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:45.114872 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" Jun 10 10:48:45.260132 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:45.260101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb"] Jun 10 10:48:45.263480 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:45.263451 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f42b9d_d00f_4058_b109_072b242bc4e0.slice/crio-321e463a0543d053482b49bdef2fe98976e12296186eb5a9f2e21ca75934e4a7 WatchSource:0}: Error finding container 321e463a0543d053482b49bdef2fe98976e12296186eb5a9f2e21ca75934e4a7: Status 404 returned error can't find the container with id 321e463a0543d053482b49bdef2fe98976e12296186eb5a9f2e21ca75934e4a7 Jun 10 10:48:45.701538 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:45.701496 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" event={"ID":"20f42b9d-d00f-4058-b109-072b242bc4e0","Type":"ContainerStarted","Data":"321e463a0543d053482b49bdef2fe98976e12296186eb5a9f2e21ca75934e4a7"} Jun 10 10:48:47.240864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.240827 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk"] Jun 10 10:48:47.243133 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.243113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" Jun 10 10:48:47.245089 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.245059 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jun 10 10:48:47.245343 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.245320 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4lwgt\"" Jun 10 10:48:47.252432 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.252400 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk"] Jun 10 10:48:47.356257 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.356223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d63ff657-91a0-4b96-ad25-87088038b125-tls-certificates\") pod \"prometheus-operator-admission-webhook-5b6b8f594-s6jfk\" (UID: \"d63ff657-91a0-4b96-ad25-87088038b125\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" Jun 10 10:48:47.457624 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.457585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d63ff657-91a0-4b96-ad25-87088038b125-tls-certificates\") pod \"prometheus-operator-admission-webhook-5b6b8f594-s6jfk\" (UID: \"d63ff657-91a0-4b96-ad25-87088038b125\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" Jun 10 10:48:47.460157 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.460137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d63ff657-91a0-4b96-ad25-87088038b125-tls-certificates\") pod \"prometheus-operator-admission-webhook-5b6b8f594-s6jfk\" (UID: \"d63ff657-91a0-4b96-ad25-87088038b125\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" Jun 10 10:48:47.553472 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.553389 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" Jun 10 10:48:47.679724 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.679679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk"] Jun 10 10:48:47.684114 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:47.684084 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63ff657_91a0_4b96_ad25_87088038b125.slice/crio-b3d3cddc256e41923f6efb906b5570fc69d9b52aa7c2745a445dcb3688c82251 WatchSource:0}: Error finding container b3d3cddc256e41923f6efb906b5570fc69d9b52aa7c2745a445dcb3688c82251: Status 404 returned error can't find the container with id b3d3cddc256e41923f6efb906b5570fc69d9b52aa7c2745a445dcb3688c82251 Jun 10 10:48:47.709833 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.709783 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" event={"ID":"20f42b9d-d00f-4058-b109-072b242bc4e0","Type":"ContainerStarted","Data":"cea820bc15c08afc80a665da1a669c649de190ab0647155a25102cb9de91ef57"} Jun 10 10:48:47.710965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.710944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" event={"ID":"d63ff657-91a0-4b96-ad25-87088038b125","Type":"ContainerStarted","Data":"b3d3cddc256e41923f6efb906b5570fc69d9b52aa7c2745a445dcb3688c82251"} Jun 10 10:48:47.726367 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:47.726312 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-74bbf69bbb-4gmkb" podStartSLOduration=34.256812034 podStartE2EDuration="35.726294858s" totalImagesPullingTime="1.469482824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:47.725471989 +0000 UTC m=+270.374527963" watchObservedRunningTime="2026-06-10 10:48:47.726294858 +0000 UTC m=+270.375350831" Jun 10 10:48:48.715788 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:48.715750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" event={"ID":"d63ff657-91a0-4b96-ad25-87088038b125","Type":"ContainerStarted","Data":"8be83a0ee6dbf15899dac43afc353825a482b1444948eb5d90592dada97420fe"} Jun 10 10:48:48.732885 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:48.732816 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" podStartSLOduration=0.857507378 podStartE2EDuration="1.732775125s" totalImagesPullingTime="875.267747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:48.731827911 +0000 UTC m=+271.380883878" watchObservedRunningTime="2026-06-10 10:48:48.732775125 +0000 UTC m=+271.381831099" Jun 10 10:48:49.719234 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:49.719199 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" Jun 10 10:48:49.723855 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:49.723823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5b6b8f594-s6jfk" Jun 10 10:48:50.299415 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.299382 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw"] Jun 10 10:48:50.302421 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.302401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.304665 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.304637 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jun 10 10:48:50.304665 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.304652 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jun 10 10:48:50.305315 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.305295 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gq29p\"" Jun 10 10:48:50.305403 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.305382 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jun 10 10:48:50.313160 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.313137 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw"] Jun 10 10:48:50.379403 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.379357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f3f3363-ad5e-4309-8eb4-d54c7479b263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.379586 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.379441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f3f3363-ad5e-4309-8eb4-d54c7479b263-prometheus-operator-tls\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.379586 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.379472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9dnq\" (UniqueName: \"kubernetes.io/projected/5f3f3363-ad5e-4309-8eb4-d54c7479b263-kube-api-access-l9dnq\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.379586 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.379504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3f3363-ad5e-4309-8eb4-d54c7479b263-metrics-client-ca\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.480408 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.480368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3f3363-ad5e-4309-8eb4-d54c7479b263-metrics-client-ca\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.480606 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.480446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f3f3363-ad5e-4309-8eb4-d54c7479b263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.480606 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.480483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f3f3363-ad5e-4309-8eb4-d54c7479b263-prometheus-operator-tls\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.480606 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.480502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9dnq\" (UniqueName: \"kubernetes.io/projected/5f3f3363-ad5e-4309-8eb4-d54c7479b263-kube-api-access-l9dnq\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.481095 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.481075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3f3363-ad5e-4309-8eb4-d54c7479b263-metrics-client-ca\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.483078 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.483043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f3f3363-ad5e-4309-8eb4-d54c7479b263-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.483170 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.483156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f3f3363-ad5e-4309-8eb4-d54c7479b263-prometheus-operator-tls\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.487937 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.487914 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9dnq\" (UniqueName: \"kubernetes.io/projected/5f3f3363-ad5e-4309-8eb4-d54c7479b263-kube-api-access-l9dnq\") pod \"prometheus-operator-7f7d445d84-tpcpw\" (UID: \"5f3f3363-ad5e-4309-8eb4-d54c7479b263\") " pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.611425 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.611383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" Jun 10 10:48:50.746391 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:50.746346 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw"] Jun 10 10:48:50.749338 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:50.749312 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3f3363_ad5e_4309_8eb4_d54c7479b263.slice/crio-a5f3ce9eb5bfdbf1ed58b25cb3ed6e9252cbe2cd69e0bb6fa4b6e00928e795fb WatchSource:0}: Error finding container a5f3ce9eb5bfdbf1ed58b25cb3ed6e9252cbe2cd69e0bb6fa4b6e00928e795fb: Status 404 returned error can't find the container with id a5f3ce9eb5bfdbf1ed58b25cb3ed6e9252cbe2cd69e0bb6fa4b6e00928e795fb Jun 10 10:48:51.727903 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:51.727852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" event={"ID":"5f3f3363-ad5e-4309-8eb4-d54c7479b263","Type":"ContainerStarted","Data":"a5f3ce9eb5bfdbf1ed58b25cb3ed6e9252cbe2cd69e0bb6fa4b6e00928e795fb"} Jun 10 10:48:52.732497 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:52.732464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" event={"ID":"5f3f3363-ad5e-4309-8eb4-d54c7479b263","Type":"ContainerStarted","Data":"a90a7d3fb2761ba99fd53fee4dbc4fc4bdb59f616cba9bc0ba7fc829173126b5"} Jun 10 10:48:52.732991 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:52.732518 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" event={"ID":"5f3f3363-ad5e-4309-8eb4-d54c7479b263","Type":"ContainerStarted","Data":"886c98cd578e18908cdaf449e2c911e52f00c785ad6c590cb95df49f0172ffc5"} Jun 10 10:48:52.750345 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:52.750282 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7f7d445d84-tpcpw" podStartSLOduration=1.678491706 podStartE2EDuration="2.750260633s" totalImagesPullingTime="1.071768927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:52.749793321 +0000 UTC m=+275.398849293" watchObservedRunningTime="2026-06-10 10:48:52.750260633 +0000 UTC m=+275.399316608" Jun 10 10:48:53.422962 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.422922 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68468f5cb9-llnr8"] Jun 10 10:48:53.426001 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.425985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.428379 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.428356 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jun 10 10:48:53.428493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.428385 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jun 10 10:48:53.428852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.428826 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jun 10 10:48:53.428960 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.428861 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jun 10 10:48:53.429179 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.429161 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jun 10 10:48:53.429179 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.429172 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jun 10 10:48:53.429334 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.429185 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qzwfh\"" Jun 10 10:48:53.429334 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.429176 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jun 10 10:48:53.434195 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.434176 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jun 10 10:48:53.437426 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.437403 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68468f5cb9-llnr8"] Jun 10 10:48:53.509631 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.509591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-trusted-ca-bundle\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.509864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.509708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-serving-cert\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.509864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.509735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-service-ca\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.509864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.509825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-kube-api-access-hrbr6\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.509864 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.509857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-config\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.510062 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.509886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-oauth-config\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.510062 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.509926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-oauth-serving-cert\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.610733 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.610691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-serving-cert\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.610733 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.610736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-service-ca\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.611008 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.610871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-kube-api-access-hrbr6\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.611008 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.610906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-config\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.611008 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.610931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-oauth-config\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.611158 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.611057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-oauth-serving-cert\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.616517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.611503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-trusted-ca-bundle\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.616517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.612216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-config\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.616517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.612374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-service-ca\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.616517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.612606 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-oauth-serving-cert\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.616517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.614902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-oauth-config\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.616517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.615324 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-serving-cert\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.616517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.615336 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-trusted-ca-bundle\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.619662 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.619634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-kube-api-access-hrbr6\") pod \"console-68468f5cb9-llnr8\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.735496 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.735395 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:48:53.867548 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:53.867521 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68468f5cb9-llnr8"] Jun 10 10:48:53.870686 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:53.870656 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd22e3ff_36dd_4d42_920f_e7e3bae9cc12.slice/crio-835b52291977c99fee3caee8f4eae6b6f3791500f4292dcd4a4326da01b4d170 WatchSource:0}: Error finding container 835b52291977c99fee3caee8f4eae6b6f3791500f4292dcd4a4326da01b4d170: Status 404 returned error can't find the container with id 835b52291977c99fee3caee8f4eae6b6f3791500f4292dcd4a4326da01b4d170 Jun 10 10:48:54.721236 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.721168 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc"] Jun 10 10:48:54.724982 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.724958 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.727287 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.727086 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jun 10 10:48:54.727429 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.727370 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jun 10 10:48:54.727959 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.727940 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dl5cn\"" Jun 10 10:48:54.743217 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.743189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68468f5cb9-llnr8" event={"ID":"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12","Type":"ContainerStarted","Data":"835b52291977c99fee3caee8f4eae6b6f3791500f4292dcd4a4326da01b4d170"} Jun 10 10:48:54.750228 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.750194 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc"] Jun 10 10:48:54.762147 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.762119 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9"] Jun 10 10:48:54.766078 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.765460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.771501 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.771473 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jun 10 10:48:54.771613 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.771496 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jun 10 10:48:54.772642 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.772619 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-52ht2\"" Jun 10 10:48:54.772865 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.772849 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jun 10 10:48:54.778931 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.778837 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zw5x6"] Jun 10 10:48:54.782232 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.782212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.784624 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.784346 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jun 10 10:48:54.784624 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.784354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jun 10 10:48:54.784772 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.784729 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-859ww\"" Jun 10 10:48:54.786190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.785140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jun 10 10:48:54.786282 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.786196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9"] Jun 10 10:48:54.824235 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.823998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9053fec-1dbf-47ea-988f-ac88247ba7f2-volume-directive-shadow\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.824235 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.824074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a190f80-8f1f-4c2b-b914-358130b54f15-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.824235 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.824123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-tls\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.824713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.824590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a190f80-8f1f-4c2b-b914-358130b54f15-openshift-state-metrics-tls\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.824896 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.824674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97sw\" (UniqueName: \"kubernetes.io/projected/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-api-access-p97sw\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.824896 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.824866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9053fec-1dbf-47ea-988f-ac88247ba7f2-metrics-client-ca\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.825126 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.825089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.825381 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.825263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a190f80-8f1f-4c2b-b914-358130b54f15-metrics-client-ca\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.825608 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.825531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.825608 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.825563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkf4\" (UniqueName: \"kubernetes.io/projected/1a190f80-8f1f-4c2b-b914-358130b54f15-kube-api-access-kqkf4\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.926858 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.926821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.926858 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.926860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a190f80-8f1f-4c2b-b914-358130b54f15-metrics-client-ca\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-root\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-tls\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkf4\" (UniqueName: \"kubernetes.io/projected/1a190f80-8f1f-4c2b-b914-358130b54f15-kube-api-access-kqkf4\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-accelerators-collector-config\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-textfile\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9053fec-1dbf-47ea-988f-ac88247ba7f2-volume-directive-shadow\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-wtmp\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqw8h\" (UniqueName: \"kubernetes.io/projected/66fdcddc-54e9-4375-9b6b-799eec109808-kube-api-access-vqw8h\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a190f80-8f1f-4c2b-b914-358130b54f15-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-sys\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a190f80-8f1f-4c2b-b914-358130b54f15-metrics-client-ca\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-tls\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.927717 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:54.927721 2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jun 10 10:48:54.928828 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.927820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a190f80-8f1f-4c2b-b914-358130b54f15-openshift-state-metrics-tls\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.928828 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.928038 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9053fec-1dbf-47ea-988f-ac88247ba7f2-volume-directive-shadow\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.928828 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:54.928274 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-tls podName:d9053fec-1dbf-47ea-988f-ac88247ba7f2 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:55.428243061 +0000 UTC m=+278.077299021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-tls") pod "kube-state-metrics-57bbf8bfb5-t2nx9" (UID: "d9053fec-1dbf-47ea-988f-ac88247ba7f2") : secret "kube-state-metrics-tls" not found Jun 10 10:48:54.928828 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.928320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97sw\" (UniqueName: \"kubernetes.io/projected/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-api-access-p97sw\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.928828 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.928354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9053fec-1dbf-47ea-988f-ac88247ba7f2-metrics-client-ca\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.928828 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.928389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66fdcddc-54e9-4375-9b6b-799eec109808-metrics-client-ca\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:54.929144 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.929085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9053fec-1dbf-47ea-988f-ac88247ba7f2-metrics-client-ca\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.930447 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.930244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.930604 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.930582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a190f80-8f1f-4c2b-b914-358130b54f15-openshift-state-metrics-tls\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.930707 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.930670 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a190f80-8f1f-4c2b-b914-358130b54f15-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:54.937155 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.937125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97sw\" (UniqueName: \"kubernetes.io/projected/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-api-access-p97sw\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:54.937267 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:54.937224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkf4\" (UniqueName: \"kubernetes.io/projected/1a190f80-8f1f-4c2b-b914-358130b54f15-kube-api-access-kqkf4\") pod \"openshift-state-metrics-65f78d5c66-pnzqc\" (UID: \"1a190f80-8f1f-4c2b-b914-358130b54f15\") " pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:55.029017 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.028929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-root\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029017 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.028970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-tls\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-root\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-accelerators-collector-config\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-textfile\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-wtmp\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqw8h\" (UniqueName: \"kubernetes.io/projected/66fdcddc-54e9-4375-9b6b-799eec109808-kube-api-access-vqw8h\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029498 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-sys\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.029498 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66fdcddc-54e9-4375-9b6b-799eec109808-metrics-client-ca\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.030036 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-accelerators-collector-config\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.030036 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.029889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-sys\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.030036 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.030006 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-wtmp\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.030263 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.030154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66fdcddc-54e9-4375-9b6b-799eec109808-metrics-client-ca\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.030263 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.030187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-textfile\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.032488 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.032467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.032696 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.032676 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66fdcddc-54e9-4375-9b6b-799eec109808-node-exporter-tls\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.037824 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.037582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" Jun 10 10:48:55.044541 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.044490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqw8h\" (UniqueName: \"kubernetes.io/projected/66fdcddc-54e9-4375-9b6b-799eec109808-kube-api-access-vqw8h\") pod \"node-exporter-zw5x6\" (UID: \"66fdcddc-54e9-4375-9b6b-799eec109808\") " pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.096812 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.096499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zw5x6" Jun 10 10:48:55.114397 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:55.114341 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fdcddc_54e9_4375_9b6b_799eec109808.slice/crio-a50b98a121208601b8ba363bfc850b568ebbd2ece0798f3032d3fdf1a384fe78 WatchSource:0}: Error finding container a50b98a121208601b8ba363bfc850b568ebbd2ece0798f3032d3fdf1a384fe78: Status 404 returned error can't find the container with id a50b98a121208601b8ba363bfc850b568ebbd2ece0798f3032d3fdf1a384fe78 Jun 10 10:48:55.209331 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.209296 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc"] Jun 10 10:48:55.213652 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:55.213621 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a190f80_8f1f_4c2b_b914_358130b54f15.slice/crio-bfd77cc7407316b8459b91963916ea1b0fe9b7e405ea3aca85e5110e177dafc9 WatchSource:0}: Error finding container bfd77cc7407316b8459b91963916ea1b0fe9b7e405ea3aca85e5110e177dafc9: Status 404 returned error can't find the container with id bfd77cc7407316b8459b91963916ea1b0fe9b7e405ea3aca85e5110e177dafc9 Jun 10 10:48:55.433345 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.433303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-tls\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:55.436303 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.436269 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9053fec-1dbf-47ea-988f-ac88247ba7f2-kube-state-metrics-tls\") pod \"kube-state-metrics-57bbf8bfb5-t2nx9\" (UID: \"d9053fec-1dbf-47ea-988f-ac88247ba7f2\") " pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:55.679637 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.679601 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" Jun 10 10:48:55.754294 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.754220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zw5x6" event={"ID":"66fdcddc-54e9-4375-9b6b-799eec109808","Type":"ContainerStarted","Data":"a50b98a121208601b8ba363bfc850b568ebbd2ece0798f3032d3fdf1a384fe78"} Jun 10 10:48:55.756851 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.756704 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" event={"ID":"1a190f80-8f1f-4c2b-b914-358130b54f15","Type":"ContainerStarted","Data":"3c08d4742814782f897f5527a22778330a63a89f7b46514c1016de17331f0f3e"} Jun 10 10:48:55.756851 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.756768 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" event={"ID":"1a190f80-8f1f-4c2b-b914-358130b54f15","Type":"ContainerStarted","Data":"40afe6878b52a7a5300da71f4d27ff47f5c327d97f559c244df20770c171f1a8"} Jun 10 10:48:55.756851 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:55.756816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" event={"ID":"1a190f80-8f1f-4c2b-b914-358130b54f15","Type":"ContainerStarted","Data":"bfd77cc7407316b8459b91963916ea1b0fe9b7e405ea3aca85e5110e177dafc9"} Jun 10 10:48:56.747041 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.747007 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8544f65558-rv8l6"] Jun 10 10:48:56.756488 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.756454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.758698 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.758679 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jun 10 10:48:56.759052 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.759031 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dfjv6\"" Jun 10 10:48:56.759194 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.759174 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jun 10 10:48:56.759194 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.759182 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jun 10 10:48:56.759315 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.759212 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jun 10 10:48:56.759315 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.759247 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8u8geuqisi9en\"" Jun 10 10:48:56.759470 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.759452 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jun 10 10:48:56.763742 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.763718 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8544f65558-rv8l6"] Jun 10 10:48:56.851629 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.851585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.851833 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.851645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.851833 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.851750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-metrics-client-ca\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.851833 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.851790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9zd\" (UniqueName: \"kubernetes.io/projected/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-kube-api-access-9h9zd\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.851997 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.851875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-grpc-tls\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.851997 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.851946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.851997 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.851975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-tls\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.852132 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.852074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953410 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953578 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953658 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-metrics-client-ca\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953715 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9zd\" (UniqueName: \"kubernetes.io/projected/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-kube-api-access-9h9zd\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953767 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-grpc-tls\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953845 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-tls\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.953955 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.953876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.955220 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.955130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-metrics-client-ca\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.957607 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.957433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-tls\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.958249 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.957855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.958249 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.958196 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.958401 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.958260 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.958838 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.958812 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-grpc-tls\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.959549 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.959517 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:56.963009 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:56.962987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9zd\" (UniqueName: \"kubernetes.io/projected/e7d6ed81-7464-47bf-9ce7-9e1c59d3e738-kube-api-access-9h9zd\") pod \"thanos-querier-8544f65558-rv8l6\" (UID: \"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738\") " pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:57.068032 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.068008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:48:57.190751 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.190724 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9"] Jun 10 10:48:57.305471 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:57.305443 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9053fec_1dbf_47ea_988f_ac88247ba7f2.slice/crio-24beb2fccb5888f34251c174e966b18e93f155f359b603524b1e59ae4db4a282 WatchSource:0}: Error finding container 24beb2fccb5888f34251c174e966b18e93f155f359b603524b1e59ae4db4a282: Status 404 returned error can't find the container with id 24beb2fccb5888f34251c174e966b18e93f155f359b603524b1e59ae4db4a282 Jun 10 10:48:57.463100 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.463071 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8544f65558-rv8l6"] Jun 10 10:48:57.504623 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:48:57.504588 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d6ed81_7464_47bf_9ce7_9e1c59d3e738.slice/crio-1ff98df9b55cd683394b316631ae2c35502659651a2495b162ba98694c11c79b WatchSource:0}: Error finding container 1ff98df9b55cd683394b316631ae2c35502659651a2495b162ba98694c11c79b: Status 404 returned error can't find the container with id 1ff98df9b55cd683394b316631ae2c35502659651a2495b162ba98694c11c79b Jun 10 10:48:57.768502 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.768320 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" event={"ID":"1a190f80-8f1f-4c2b-b914-358130b54f15","Type":"ContainerStarted","Data":"4966fa16eb6c1d9254c8831c46d11b9e4eb3121489800ed80704b13cc0c663fa"} Jun 10 10:48:57.772893 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.772835 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" event={"ID":"d9053fec-1dbf-47ea-988f-ac88247ba7f2","Type":"ContainerStarted","Data":"24beb2fccb5888f34251c174e966b18e93f155f359b603524b1e59ae4db4a282"} Jun 10 10:48:57.774929 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.774886 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68468f5cb9-llnr8" event={"ID":"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12","Type":"ContainerStarted","Data":"20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33"} Jun 10 10:48:57.776921 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.776900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" event={"ID":"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738","Type":"ContainerStarted","Data":"1ff98df9b55cd683394b316631ae2c35502659651a2495b162ba98694c11c79b"} Jun 10 10:48:57.779965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.779603 2578 generic.go:356] "Generic (PLEG): container finished" podID="66fdcddc-54e9-4375-9b6b-799eec109808" containerID="04f0fa917638edd04bd3797f2d21c2a1b6239a6db2e4b34345b95fdfb43d18b7" exitCode=0 Jun 10 10:48:57.779965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.779646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zw5x6" event={"ID":"66fdcddc-54e9-4375-9b6b-799eec109808","Type":"ContainerDied","Data":"04f0fa917638edd04bd3797f2d21c2a1b6239a6db2e4b34345b95fdfb43d18b7"} Jun 10 10:48:57.799818 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.799510 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-65f78d5c66-pnzqc" podStartSLOduration=1.801709181 podStartE2EDuration="3.799490342s" totalImagesPullingTime="1.997781161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:57.798391477 +0000 UTC m=+280.447447454" watchObservedRunningTime="2026-06-10 10:48:57.799490342 +0000 UTC m=+280.448546306" Jun 10 10:48:57.823421 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:57.823315 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68468f5cb9-llnr8" podStartSLOduration=1.637684208 podStartE2EDuration="4.823290405s" totalImagesPullingTime="3.185606197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:57.823021026 +0000 UTC m=+280.472076998" watchObservedRunningTime="2026-06-10 10:48:57.823290405 +0000 UTC m=+280.472346382" Jun 10 10:48:58.359078 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:58.359006 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vg7kl" podUID="cacd2e06-fcd6-43b0-aff9-6f114eddd501" Jun 10 10:48:58.359253 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:48:58.359075 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" Jun 10 10:48:58.788221 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:58.788178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zw5x6" event={"ID":"66fdcddc-54e9-4375-9b6b-799eec109808","Type":"ContainerStarted","Data":"48319d5cd02c52dfa63a8de77c8bc3f96c871859af7281209db03cfaa43a1014"} Jun 10 10:48:58.788688 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:58.788269 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zw5x6" event={"ID":"66fdcddc-54e9-4375-9b6b-799eec109808","Type":"ContainerStarted","Data":"f8361c9b14ab83a5e61250600f342a8adacc9f530411469c7c8233ffc79cb892"} Jun 10 10:48:58.790411 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:58.790349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" event={"ID":"d9053fec-1dbf-47ea-988f-ac88247ba7f2","Type":"ContainerStarted","Data":"1b555f2555a69926c4a5e26db6e9e6387f63281a6e1d09c7c79b9fdb706ec977"} Jun 10 10:48:58.790559 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:58.790530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vg7kl" Jun 10 10:48:58.791026 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:58.790711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:48:58.812102 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:58.812049 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zw5x6" podStartSLOduration=2.781669744 podStartE2EDuration="4.81203069s" totalImagesPullingTime="1.937814164s" totalInitContainerRuntime="92.546782ms" isStatefulPod=true podCreationTimestamp="2026-06-10 10:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:58.810277066 +0000 UTC m=+281.459333042" watchObservedRunningTime="2026-06-10 10:48:58.81203069 +0000 UTC m=+281.461086663" Jun 10 10:48:59.495651 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.495614 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f9987bf5d-h6r77"] Jun 10 10:48:59.499623 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.499597 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.511553 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.511518 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9987bf5d-h6r77"] Jun 10 10:48:59.577065 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.577033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-console-config\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.577245 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.577073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-oauth-serving-cert\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.577245 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.577143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-service-ca\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.577245 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.577196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-serving-cert\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.577245 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.577215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-trusted-ca-bundle\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.577373 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.577277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnww8\" (UniqueName: \"kubernetes.io/projected/07b21dc7-8429-41df-9aa0-9072b77cce07-kube-api-access-gnww8\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.577373 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.577298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-oauth-config\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.678452 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.678410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-serving-cert\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.678452 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.678452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-trusted-ca-bundle\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.678700 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.678490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnww8\" (UniqueName: \"kubernetes.io/projected/07b21dc7-8429-41df-9aa0-9072b77cce07-kube-api-access-gnww8\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.678700 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.678508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-oauth-config\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.678700 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.678559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-console-config\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.678700 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.678585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-oauth-serving-cert\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.678700 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.678626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-service-ca\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.679464 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.679421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-console-config\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.679575 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.679462 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-trusted-ca-bundle\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.679681 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.679657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-service-ca\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.679835 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.679788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-oauth-serving-cert\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.681177 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.681153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-serving-cert\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.681267 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.681251 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-oauth-config\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.687427 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.687409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnww8\" (UniqueName: \"kubernetes.io/projected/07b21dc7-8429-41df-9aa0-9072b77cce07-kube-api-access-gnww8\") pod \"console-f9987bf5d-h6r77\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.795999 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.795912 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" event={"ID":"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738","Type":"ContainerStarted","Data":"6f52221ca2bb0683d8789a9940329d82201235331f4b886242c01bbd4764f32a"} Jun 10 10:48:59.795999 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.795980 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" event={"ID":"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738","Type":"ContainerStarted","Data":"2fb06d50a35f2871f23e349a0551b3fac8543820089260fd1938fd655bc427ec"} Jun 10 10:48:59.796450 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.796019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" event={"ID":"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738","Type":"ContainerStarted","Data":"cc4742f90fed16f64b7e53a5c4be60dc3e3ddfd0c7ab37341b9faa0487f19be6"} Jun 10 10:48:59.797859 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.797832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" event={"ID":"d9053fec-1dbf-47ea-988f-ac88247ba7f2","Type":"ContainerStarted","Data":"257a0aeb205884d71d9362e4aeaa97302f2a62f92457209d1c0e42dc3e659efd"} Jun 10 10:48:59.797965 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.797881 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" event={"ID":"d9053fec-1dbf-47ea-988f-ac88247ba7f2","Type":"ContainerStarted","Data":"88e6e99d813762eedc321e7d62d4ceabd2f7732409c8e9d7ec69027f13a235c8"} Jun 10 10:48:59.812235 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.812196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:48:59.816498 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.816453 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-57bbf8bfb5-t2nx9" podStartSLOduration=4.547843104 podStartE2EDuration="5.816436269s" totalImagesPullingTime="1.268593165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:59.815418783 +0000 UTC m=+282.464474758" watchObservedRunningTime="2026-06-10 10:48:59.816436269 +0000 UTC m=+282.465492243" Jun 10 10:48:59.982658 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:48:59.982626 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9987bf5d-h6r77"] Jun 10 10:49:00.097429 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:49:00.097393 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b21dc7_8429_41df_9aa0_9072b77cce07.slice/crio-b8b10f020784b53ca7b16ed0ebad838e49836d8dc629fb275a2d13a144cf8324 WatchSource:0}: Error finding container b8b10f020784b53ca7b16ed0ebad838e49836d8dc629fb275a2d13a144cf8324: Status 404 returned error can't find the container with id b8b10f020784b53ca7b16ed0ebad838e49836d8dc629fb275a2d13a144cf8324 Jun 10 10:49:00.805407 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:00.805305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9987bf5d-h6r77" event={"ID":"07b21dc7-8429-41df-9aa0-9072b77cce07","Type":"ContainerStarted","Data":"b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5"} Jun 10 10:49:00.805407 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:00.805362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9987bf5d-h6r77" event={"ID":"07b21dc7-8429-41df-9aa0-9072b77cce07","Type":"ContainerStarted","Data":"b8b10f020784b53ca7b16ed0ebad838e49836d8dc629fb275a2d13a144cf8324"} Jun 10 10:49:00.809107 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:00.809075 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" event={"ID":"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738","Type":"ContainerStarted","Data":"18da9e538fd1a706b4d4bc21d9282d2b2f6f235b6add24291f7a121df6ecfb2f"} Jun 10 10:49:00.809293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:00.809148 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" event={"ID":"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738","Type":"ContainerStarted","Data":"26a0c7f8915fb580a9b8cd8f5028be32b75bbd15cbfb0bc6a8172a42f4895df8"} Jun 10 10:49:00.809293 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:00.809186 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" event={"ID":"e7d6ed81-7464-47bf-9ce7-9e1c59d3e738","Type":"ContainerStarted","Data":"f984d3212c3f4867952189fccca023dd8bf347ad2eb736064080c25fb9775870"} Jun 10 10:49:00.827782 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:00.827715 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f9987bf5d-h6r77" podStartSLOduration=1.827684219 podStartE2EDuration="1.827684219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:00.826582469 +0000 UTC m=+283.475638464" watchObservedRunningTime="2026-06-10 10:49:00.827684219 +0000 UTC m=+283.476740193" Jun 10 10:49:00.858221 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:00.858162 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" podStartSLOduration=2.445129075 podStartE2EDuration="4.858143551s" totalImagesPullingTime="2.413014476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:00.855909678 +0000 UTC m=+283.504965653" watchObservedRunningTime="2026-06-10 10:49:00.858143551 +0000 UTC m=+283.507199540" Jun 10 10:49:01.701018 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.700989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:49:01.701189 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.701079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:49:01.703525 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.703499 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cacd2e06-fcd6-43b0-aff9-6f114eddd501-metrics-tls\") pod \"dns-default-vg7kl\" (UID: \"cacd2e06-fcd6-43b0-aff9-6f114eddd501\") " pod="openshift-dns/dns-default-vg7kl" Jun 10 10:49:01.703525 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.703520 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"image-registry-7b6f49bc56-wpzlc\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:49:01.794560 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.794528 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-977dh\"" Jun 10 10:49:01.795052 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.794828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6lbht\"" Jun 10 10:49:01.801640 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.801603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:49:01.801789 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.801751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vg7kl" Jun 10 10:49:01.814390 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.814365 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:49:01.983114 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.983084 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b6f49bc56-wpzlc"] Jun 10 10:49:01.985517 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:49:01.985489 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3da4c40_3de7_4f9b_80c5_08d9042085e8.slice/crio-cc9a0f0e7720b243af4de3ccaef90766f15c6311910f6a3ed115c3f32192396c WatchSource:0}: Error finding container cc9a0f0e7720b243af4de3ccaef90766f15c6311910f6a3ed115c3f32192396c: Status 404 returned error can't find the container with id cc9a0f0e7720b243af4de3ccaef90766f15c6311910f6a3ed115c3f32192396c Jun 10 10:49:01.991476 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:01.991448 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vg7kl"] Jun 10 10:49:01.993594 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:49:01.993570 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd2e06_fcd6_43b0_aff9_6f114eddd501.slice/crio-626ad22a305a6a8ca95cb59f8b485996b320d8d3ec7af94c989144a4df2cb1a3 WatchSource:0}: Error finding container 626ad22a305a6a8ca95cb59f8b485996b320d8d3ec7af94c989144a4df2cb1a3: Status 404 returned error can't find the container with id 626ad22a305a6a8ca95cb59f8b485996b320d8d3ec7af94c989144a4df2cb1a3 Jun 10 10:49:02.820881 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:02.820789 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" event={"ID":"b3da4c40-3de7-4f9b-80c5-08d9042085e8","Type":"ContainerStarted","Data":"b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff"} Jun 10 10:49:02.821349 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:02.820907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" event={"ID":"b3da4c40-3de7-4f9b-80c5-08d9042085e8","Type":"ContainerStarted","Data":"cc9a0f0e7720b243af4de3ccaef90766f15c6311910f6a3ed115c3f32192396c"} Jun 10 10:49:02.821349 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:02.820960 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:49:02.822185 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:02.822164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vg7kl" event={"ID":"cacd2e06-fcd6-43b0-aff9-6f114eddd501","Type":"ContainerStarted","Data":"626ad22a305a6a8ca95cb59f8b485996b320d8d3ec7af94c989144a4df2cb1a3"} Jun 10 10:49:02.856493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:02.856434 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" podStartSLOduration=284.856416696 podStartE2EDuration="4m44.856416696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:02.855047547 +0000 UTC m=+285.504103521" watchObservedRunningTime="2026-06-10 10:49:02.856416696 +0000 UTC m=+285.505472673" Jun 10 10:49:03.736503 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.736409 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:49:03.736503 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.736476 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:49:03.737947 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.737924 2578 patch_prober.go:28] interesting pod/console-68468f5cb9-llnr8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Jun 10 10:49:03.738052 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.737971 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-68468f5cb9-llnr8" podUID="dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" containerName="console" probeResult="failure" output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" Jun 10 10:49:03.827540 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.827497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vg7kl" event={"ID":"cacd2e06-fcd6-43b0-aff9-6f114eddd501","Type":"ContainerStarted","Data":"69c9479657552ae78da5f5b717d582a239cd51f6ddf05fab83b42cbda2c82656"} Jun 10 10:49:03.827950 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.827561 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vg7kl" event={"ID":"cacd2e06-fcd6-43b0-aff9-6f114eddd501","Type":"ContainerStarted","Data":"1b150762fceea7b5085d02368c0bb6196d5ff4d494a04be4ab9a39639c0355d6"} Jun 10 10:49:03.827950 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.827827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vg7kl" Jun 10 10:49:03.848557 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:03.848502 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vg7kl" podStartSLOduration=251.399395077 podStartE2EDuration="4m12.848486612s" totalImagesPullingTime="1.449091535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:03.846865294 +0000 UTC m=+286.495921266" watchObservedRunningTime="2026-06-10 10:49:03.848486612 +0000 UTC m=+286.497542585" Jun 10 10:49:06.240787 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.240742 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68468f5cb9-llnr8"] Jun 10 10:49:06.270869 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.270839 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f9b5c64df-bkjpk"] Jun 10 10:49:06.274476 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.274454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.285223 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.285200 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f9b5c64df-bkjpk"] Jun 10 10:49:06.443021 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.442980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-serving-cert\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.443021 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.443022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-trusted-ca-bundle\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.443275 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.443081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-oauth-serving-cert\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.443275 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.443135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-service-ca\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.443275 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.443218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-oauth-config\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.443275 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.443250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcd4m\" (UniqueName: \"kubernetes.io/projected/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-kube-api-access-qcd4m\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.443446 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.443285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-config\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.544479 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.544340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-oauth-config\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.544479 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.544391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcd4m\" (UniqueName: \"kubernetes.io/projected/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-kube-api-access-qcd4m\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.544479 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.544425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-config\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.544768 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.544504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-serving-cert\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.544768 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.544539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-trusted-ca-bundle\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.544768 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.544570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-oauth-serving-cert\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.544768 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.544604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-service-ca\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.545400 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.545376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-config\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.545400 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.545388 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-service-ca\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.545719 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.545701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-trusted-ca-bundle\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.545758 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.545716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-oauth-serving-cert\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.547096 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.547075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-oauth-config\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.547253 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.547234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-serving-cert\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.552304 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.552283 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcd4m\" (UniqueName: \"kubernetes.io/projected/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-kube-api-access-qcd4m\") pod \"console-6f9b5c64df-bkjpk\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.583891 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.583852 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:06.716897 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.716848 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f9b5c64df-bkjpk"] Jun 10 10:49:06.718988 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:49:06.718958 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43035f7c_dbe5_4176_8a2b_e8b6763e23a1.slice/crio-76a5cc30799ee54b8e273b9150386f83a634b31c919e3a7909f767ea8bea7c44 WatchSource:0}: Error finding container 76a5cc30799ee54b8e273b9150386f83a634b31c919e3a7909f767ea8bea7c44: Status 404 returned error can't find the container with id 76a5cc30799ee54b8e273b9150386f83a634b31c919e3a7909f767ea8bea7c44 Jun 10 10:49:06.843689 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.843658 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f9b5c64df-bkjpk" event={"ID":"43035f7c-dbe5-4176-8a2b-e8b6763e23a1","Type":"ContainerStarted","Data":"467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85"} Jun 10 10:49:06.843893 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:06.843714 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f9b5c64df-bkjpk" event={"ID":"43035f7c-dbe5-4176-8a2b-e8b6763e23a1","Type":"ContainerStarted","Data":"76a5cc30799ee54b8e273b9150386f83a634b31c919e3a7909f767ea8bea7c44"} Jun 10 10:49:07.828786 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:07.828752 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8544f65558-rv8l6" Jun 10 10:49:07.854191 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:07.854138 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f9b5c64df-bkjpk" podStartSLOduration=1.854121463 podStartE2EDuration="1.854121463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:06.864226907 +0000 UTC m=+289.513282881" watchObservedRunningTime="2026-06-10 10:49:07.854121463 +0000 UTC m=+290.503177470" Jun 10 10:49:09.813037 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:09.813003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:49:09.813407 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:09.813078 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:49:09.814391 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:09.814369 2578 patch_prober.go:28] interesting pod/console-f9987bf5d-h6r77 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.24:8443/health\": dial tcp 10.132.0.24:8443: connect: connection refused" start-of-body= Jun 10 10:49:09.814493 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:09.814408 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-f9987bf5d-h6r77" podUID="07b21dc7-8429-41df-9aa0-9072b77cce07" containerName="console" probeResult="failure" output="Get \"https://10.132.0.24:8443/health\": dial tcp 10.132.0.24:8443: connect: connection refused" Jun 10 10:49:09.850383 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:49:09.850352 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8t72g" podUID="8327523a-eb26-4c9d-847a-761b868b2d27" Jun 10 10:49:13.834038 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:13.834003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vg7kl" Jun 10 10:49:16.584994 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:16.584950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:16.585417 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:16.585014 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:16.586706 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:16.586680 2578 patch_prober.go:28] interesting pod/console-6f9b5c64df-bkjpk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.25:8443/health\": dial tcp 10.132.0.25:8443: connect: connection refused" start-of-body= Jun 10 10:49:16.586862 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:16.586726 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6f9b5c64df-bkjpk" podUID="43035f7c-dbe5-4176-8a2b-e8b6763e23a1" containerName="console" probeResult="failure" output="Get \"https://10.132.0.25:8443/health\": dial tcp 10.132.0.25:8443: connect: connection refused" Jun 10 10:49:17.795272 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:17.795239 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:49:17.796613 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:17.796593 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:49:17.803125 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:17.803104 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:49:17.804521 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:17.804492 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:49:17.807980 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:17.807961 2578 kubelet.go:1648] "Image garbage collection succeeded" Jun 10 10:49:19.816716 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:19.816689 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:49:19.839473 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:19.820364 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:49:21.805856 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:21.805816 2578 patch_prober.go:28] interesting pod/image-registry-7b6f49bc56-wpzlc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 10 10:49:21.806214 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:21.805880 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 10 10:49:23.832016 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:23.831984 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:49:24.849389 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:24.849349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:49:26.588749 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:26.588718 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:26.592708 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:26.592684 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:49:26.655022 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:26.654989 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9987bf5d-h6r77"] Jun 10 10:49:29.246197 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:29.246163 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b6f49bc56-wpzlc"] Jun 10 10:49:31.265792 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.265724 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-68468f5cb9-llnr8" podUID="dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" containerName="console" containerID="cri-o://20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33" gracePeriod=15 Jun 10 10:49:31.519318 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.519256 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:49:31.680689 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.680650 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-oauth-config\") pod \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " Jun 10 10:49:31.680910 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.680703 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-trusted-ca-bundle\") pod \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " Jun 10 10:49:31.680910 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.680733 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-service-ca\") pod \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " Jun 10 10:49:31.680910 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.680775 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-serving-cert\") pod \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " Jun 10 10:49:31.680910 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.680791 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-oauth-serving-cert\") pod \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " Jun 10 10:49:31.681202 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.680912 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-kube-api-access-hrbr6\") pod \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " Jun 10 10:49:31.681202 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.680983 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-config\") pod \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\" (UID: \"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12\") " Jun 10 10:49:31.681310 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.681228 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" (UID: "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:31.681310 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.681232 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" (UID: "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:31.681310 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.681245 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-service-ca" (OuterVolumeSpecName: "service-ca") pod "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" (UID: "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:31.681429 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.681322 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-oauth-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:31.681429 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.681334 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-trusted-ca-bundle\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:31.681429 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.681342 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-service-ca\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:31.681733 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.681709 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-config" (OuterVolumeSpecName: "console-config") pod "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" (UID: "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:31.683176 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.683151 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" (UID: "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:31.683641 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.683616 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" (UID: "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:31.683641 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.683631 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-kube-api-access-hrbr6" (OuterVolumeSpecName: "kube-api-access-hrbr6") pod "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" (UID: "dd22e3ff-36dd-4d42-920f-e7e3bae9cc12"). InnerVolumeSpecName "kube-api-access-hrbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:49:31.781936 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.781846 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:31.781936 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.781876 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-kube-api-access-hrbr6\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:31.781936 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.781886 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:31.781936 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.781895 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12-console-oauth-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:31.953000 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.952965 2578 generic.go:356] "Generic (PLEG): container finished" podID="dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" containerID="20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33" exitCode=0 Jun 10 10:49:31.953187 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.953007 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68468f5cb9-llnr8" Jun 10 10:49:31.953187 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.953060 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68468f5cb9-llnr8" event={"ID":"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12","Type":"ContainerDied","Data":"20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33"} Jun 10 10:49:31.953187 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.953112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68468f5cb9-llnr8" event={"ID":"dd22e3ff-36dd-4d42-920f-e7e3bae9cc12","Type":"ContainerDied","Data":"835b52291977c99fee3caee8f4eae6b6f3791500f4292dcd4a4326da01b4d170"} Jun 10 10:49:31.953187 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.953137 2578 scope.go:117] "RemoveContainer" containerID="20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33" Jun 10 10:49:31.961524 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.961504 2578 scope.go:117] "RemoveContainer" containerID="20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33" Jun 10 10:49:31.961846 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:49:31.961824 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33\": container with ID starting with 20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33 not found: ID does not exist" containerID="20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33" Jun 10 10:49:31.961938 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.961854 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33"} err="failed to get container status \"20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33\": rpc error: code = NotFound desc = could not find container \"20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33\": container with ID starting with 20a93980b852bcdf331bcd4684d6276419735487c38f461cde01dc23a7b83a33 not found: ID does not exist" Jun 10 10:49:31.970084 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.970059 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68468f5cb9-llnr8"] Jun 10 10:49:31.972304 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:31.972279 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68468f5cb9-llnr8"] Jun 10 10:49:32.959563 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:32.959530 2578 generic.go:356] "Generic (PLEG): container finished" podID="4075f9cd-34d0-4601-b9ed-227e3918826e" containerID="0f92d7d2e0c4796013e1e88b31ea51139f12a0250bedabe552e0ab8e755a344f" exitCode=0 Jun 10 10:49:32.960088 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:32.959612 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" event={"ID":"4075f9cd-34d0-4601-b9ed-227e3918826e","Type":"ContainerDied","Data":"0f92d7d2e0c4796013e1e88b31ea51139f12a0250bedabe552e0ab8e755a344f"} Jun 10 10:49:32.960088 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:32.960043 2578 scope.go:117] "RemoveContainer" containerID="0f92d7d2e0c4796013e1e88b31ea51139f12a0250bedabe552e0ab8e755a344f" Jun 10 10:49:33.857222 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:33.857189 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" path="/var/lib/kubelet/pods/dd22e3ff-36dd-4d42-920f-e7e3bae9cc12/volumes" Jun 10 10:49:33.965998 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:33.965967 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55bb7c4d49-pszhc" event={"ID":"4075f9cd-34d0-4601-b9ed-227e3918826e","Type":"ContainerStarted","Data":"e2e400767db0fca1457878e9560d972497cc48b6bc3ae6bbd076ccc03be09627"} Jun 10 10:49:48.022307 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:48.022219 2578 generic.go:356] "Generic (PLEG): container finished" podID="dfc1a540-76c8-4702-be98-ba0d4a1759f3" containerID="b900d3671734e5f0b89709961cdcf2b85cc039d88690b184a246e91338794fc6" exitCode=0 Jun 10 10:49:48.022307 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:48.022264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-rjfdv" event={"ID":"dfc1a540-76c8-4702-be98-ba0d4a1759f3","Type":"ContainerDied","Data":"b900d3671734e5f0b89709961cdcf2b85cc039d88690b184a246e91338794fc6"} Jun 10 10:49:48.022738 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:48.022601 2578 scope.go:117] "RemoveContainer" containerID="b900d3671734e5f0b89709961cdcf2b85cc039d88690b184a246e91338794fc6" Jun 10 10:49:49.028237 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:49.028206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-rjfdv" event={"ID":"dfc1a540-76c8-4702-be98-ba0d4a1759f3","Type":"ContainerStarted","Data":"48bf81d328fc434a631ba76bc0d8a4ed2ea377b540f3646aebc0b478caf138bd"} Jun 10 10:49:51.680654 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.680607 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-f9987bf5d-h6r77" podUID="07b21dc7-8429-41df-9aa0-9072b77cce07" containerName="console" containerID="cri-o://b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5" gracePeriod=15 Jun 10 10:49:51.934051 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.933986 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:49:51.967026 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.966993 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnww8\" (UniqueName: \"kubernetes.io/projected/07b21dc7-8429-41df-9aa0-9072b77cce07-kube-api-access-gnww8\") pod \"07b21dc7-8429-41df-9aa0-9072b77cce07\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " Jun 10 10:49:51.967026 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.967033 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-serving-cert\") pod \"07b21dc7-8429-41df-9aa0-9072b77cce07\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " Jun 10 10:49:51.967268 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.967076 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-service-ca\") pod \"07b21dc7-8429-41df-9aa0-9072b77cce07\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " Jun 10 10:49:51.967268 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.967110 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-oauth-config\") pod \"07b21dc7-8429-41df-9aa0-9072b77cce07\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " Jun 10 10:49:51.967268 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.967132 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-oauth-serving-cert\") pod \"07b21dc7-8429-41df-9aa0-9072b77cce07\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " Jun 10 10:49:51.967268 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.967158 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-trusted-ca-bundle\") pod \"07b21dc7-8429-41df-9aa0-9072b77cce07\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " Jun 10 10:49:51.967268 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.967201 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-console-config\") pod \"07b21dc7-8429-41df-9aa0-9072b77cce07\" (UID: \"07b21dc7-8429-41df-9aa0-9072b77cce07\") " Jun 10 10:49:51.968321 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.967934 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-service-ca" (OuterVolumeSpecName: "service-ca") pod "07b21dc7-8429-41df-9aa0-9072b77cce07" (UID: "07b21dc7-8429-41df-9aa0-9072b77cce07"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:51.968321 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.968042 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "07b21dc7-8429-41df-9aa0-9072b77cce07" (UID: "07b21dc7-8429-41df-9aa0-9072b77cce07"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:51.968321 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.968115 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-console-config" (OuterVolumeSpecName: "console-config") pod "07b21dc7-8429-41df-9aa0-9072b77cce07" (UID: "07b21dc7-8429-41df-9aa0-9072b77cce07"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:51.968321 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.968279 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "07b21dc7-8429-41df-9aa0-9072b77cce07" (UID: "07b21dc7-8429-41df-9aa0-9072b77cce07"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:51.969983 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.968836 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-service-ca\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:51.969983 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.968864 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-oauth-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:51.969983 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.968879 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-trusted-ca-bundle\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:51.969983 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.968899 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07b21dc7-8429-41df-9aa0-9072b77cce07-console-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:51.971843 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.970341 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "07b21dc7-8429-41df-9aa0-9072b77cce07" (UID: "07b21dc7-8429-41df-9aa0-9072b77cce07"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:51.975995 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.975947 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b21dc7-8429-41df-9aa0-9072b77cce07-kube-api-access-gnww8" (OuterVolumeSpecName: "kube-api-access-gnww8") pod "07b21dc7-8429-41df-9aa0-9072b77cce07" (UID: "07b21dc7-8429-41df-9aa0-9072b77cce07"). InnerVolumeSpecName "kube-api-access-gnww8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:49:51.975995 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:51.975825 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "07b21dc7-8429-41df-9aa0-9072b77cce07" (UID: "07b21dc7-8429-41df-9aa0-9072b77cce07"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:52.041712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.041677 2578 generic.go:356] "Generic (PLEG): container finished" podID="07b21dc7-8429-41df-9aa0-9072b77cce07" containerID="b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5" exitCode=0 Jun 10 10:49:52.041915 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.041726 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9987bf5d-h6r77" event={"ID":"07b21dc7-8429-41df-9aa0-9072b77cce07","Type":"ContainerDied","Data":"b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5"} Jun 10 10:49:52.041915 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.041743 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9987bf5d-h6r77" Jun 10 10:49:52.041915 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.041827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9987bf5d-h6r77" event={"ID":"07b21dc7-8429-41df-9aa0-9072b77cce07","Type":"ContainerDied","Data":"b8b10f020784b53ca7b16ed0ebad838e49836d8dc629fb275a2d13a144cf8324"} Jun 10 10:49:52.041915 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.041863 2578 scope.go:117] "RemoveContainer" containerID="b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5" Jun 10 10:49:52.051230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.051207 2578 scope.go:117] "RemoveContainer" containerID="b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5" Jun 10 10:49:52.051542 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:49:52.051522 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5\": container with ID starting with b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5 not found: ID does not exist" containerID="b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5" Jun 10 10:49:52.051616 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.051554 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5"} err="failed to get container status \"b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5\": rpc error: code = NotFound desc = could not find container \"b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5\": container with ID starting with b60af0e64070a33df2adcb87930c901c16b12dc7b5c61599f123524f31d601e5 not found: ID does not exist" Jun 10 10:49:52.062509 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.062473 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9987bf5d-h6r77"] Jun 10 10:49:52.065928 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.065894 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f9987bf5d-h6r77"] Jun 10 10:49:52.069611 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.069585 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-oauth-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:52.069611 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.069609 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnww8\" (UniqueName: \"kubernetes.io/projected/07b21dc7-8429-41df-9aa0-9072b77cce07-kube-api-access-gnww8\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:52.069791 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:52.069620 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07b21dc7-8429-41df-9aa0-9072b77cce07-console-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:53.853812 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:53.853759 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b21dc7-8429-41df-9aa0-9072b77cce07" path="/var/lib/kubelet/pods/07b21dc7-8429-41df-9aa0-9072b77cce07/volumes" Jun 10 10:49:54.267581 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:54.267462 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" containerName="registry" containerID="cri-o://b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff" gracePeriod=30 Jun 10 10:49:55.511156 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.511123 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:49:55.601298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601195 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3da4c40-3de7-4f9b-80c5-08d9042085e8-ca-trust-extracted\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601257 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-trusted-ca\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601297 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601582 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601334 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk2dg\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-kube-api-access-rk2dg\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601582 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601383 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-bound-sa-token\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601582 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601414 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-certificates\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601582 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-installation-pull-secrets\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601582 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601498 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-image-registry-private-configuration\") pod \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\" (UID: \"b3da4c40-3de7-4f9b-80c5-08d9042085e8\") " Jun 10 10:49:55.601958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.601908 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:55.602243 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.602079 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:55.603847 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.603788 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:49:55.604174 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.604151 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:55.604279 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.604262 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-kube-api-access-rk2dg" (OuterVolumeSpecName: "kube-api-access-rk2dg") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "kube-api-access-rk2dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:49:55.604481 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.604450 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:49:55.604586 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.604511 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:55.610074 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.610041 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da4c40-3de7-4f9b-80c5-08d9042085e8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b3da4c40-3de7-4f9b-80c5-08d9042085e8" (UID: "b3da4c40-3de7-4f9b-80c5-08d9042085e8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 10:49:55.702849 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702780 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-certificates\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:55.702849 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702844 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-installation-pull-secrets\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:55.702849 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702856 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3da4c40-3de7-4f9b-80c5-08d9042085e8-image-registry-private-configuration\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:55.703104 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702866 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3da4c40-3de7-4f9b-80c5-08d9042085e8-ca-trust-extracted\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:55.703104 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702874 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3da4c40-3de7-4f9b-80c5-08d9042085e8-trusted-ca\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:55.703104 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702886 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:55.703104 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702894 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2dg\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-kube-api-access-rk2dg\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:55.703104 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:55.702902 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3da4c40-3de7-4f9b-80c5-08d9042085e8-bound-sa-token\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:49:56.059593 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.059558 2578 generic.go:356] "Generic (PLEG): container finished" podID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" containerID="b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff" exitCode=0 Jun 10 10:49:56.059593 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.059602 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" event={"ID":"b3da4c40-3de7-4f9b-80c5-08d9042085e8","Type":"ContainerDied","Data":"b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff"} Jun 10 10:49:56.059593 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.059605 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" Jun 10 10:49:56.059870 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.059647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b6f49bc56-wpzlc" event={"ID":"b3da4c40-3de7-4f9b-80c5-08d9042085e8","Type":"ContainerDied","Data":"cc9a0f0e7720b243af4de3ccaef90766f15c6311910f6a3ed115c3f32192396c"} Jun 10 10:49:56.059870 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.059670 2578 scope.go:117] "RemoveContainer" containerID="b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff" Jun 10 10:49:56.070151 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.070123 2578 scope.go:117] "RemoveContainer" containerID="b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff" Jun 10 10:49:56.070742 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:49:56.070719 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff\": container with ID starting with b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff not found: ID does not exist" containerID="b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff" Jun 10 10:49:56.070788 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.070751 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff"} err="failed to get container status \"b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff\": rpc error: code = NotFound desc = could not find container \"b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff\": container with ID starting with b0582e0bff96837ef3f3d4c38d66101a6694f83a06d884620e6f7a2b8b84ceff not found: ID does not exist" Jun 10 10:49:56.074667 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.074641 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b6f49bc56-wpzlc"] Jun 10 10:49:56.078958 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:56.078934 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b6f49bc56-wpzlc"] Jun 10 10:49:57.854397 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:49:57.854351 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" path="/var/lib/kubelet/pods/b3da4c40-3de7-4f9b-80c5-08d9042085e8/volumes" Jun 10 10:50:16.196449 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196356 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68c4548f6b-xdqfc"] Jun 10 10:50:16.196903 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196728 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" containerName="console" Jun 10 10:50:16.196903 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196745 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" containerName="console" Jun 10 10:50:16.196903 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196761 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" containerName="registry" Jun 10 10:50:16.196903 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196770 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" containerName="registry" Jun 10 10:50:16.196903 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196810 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b21dc7-8429-41df-9aa0-9072b77cce07" containerName="console" Jun 10 10:50:16.196903 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196820 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b21dc7-8429-41df-9aa0-9072b77cce07" containerName="console" Jun 10 10:50:16.197136 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196917 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd22e3ff-36dd-4d42-920f-e7e3bae9cc12" containerName="console" Jun 10 10:50:16.197136 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196930 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b21dc7-8429-41df-9aa0-9072b77cce07" containerName="console" Jun 10 10:50:16.197136 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.196941 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3da4c40-3de7-4f9b-80c5-08d9042085e8" containerName="registry" Jun 10 10:50:16.200139 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.200114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.213532 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.213509 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c4548f6b-xdqfc"] Jun 10 10:50:16.278679 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.278635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4x8\" (UniqueName: \"kubernetes.io/projected/7ad03660-73d0-4233-8689-04ccbbd84555-kube-api-access-8l4x8\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.278679 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.278682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-oauth-serving-cert\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.278969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.278716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-service-ca\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.278969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.278741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-trusted-ca-bundle\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.278969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.278773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-console-config\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.278969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.278858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-serving-cert\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.278969 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.278922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-oauth-config\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.380190 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.380158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4x8\" (UniqueName: \"kubernetes.io/projected/7ad03660-73d0-4233-8689-04ccbbd84555-kube-api-access-8l4x8\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.380380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.380196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-oauth-serving-cert\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.380380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.380218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-service-ca\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.380380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.380243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-trusted-ca-bundle\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.380380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.380279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-console-config\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.380380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.380322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-serving-cert\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.380380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.380380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-oauth-config\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.381057 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.381030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-service-ca\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.381155 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.381133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-console-config\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.381189 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.381132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-oauth-serving-cert\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.381226 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.381199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-trusted-ca-bundle\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.383055 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.383029 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-serving-cert\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.383215 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.383197 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-oauth-config\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.394229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.394198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4x8\" (UniqueName: \"kubernetes.io/projected/7ad03660-73d0-4233-8689-04ccbbd84555-kube-api-access-8l4x8\") pod \"console-68c4548f6b-xdqfc\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.510477 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.510375 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:16.648220 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:16.648187 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c4548f6b-xdqfc"] Jun 10 10:50:16.651551 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:50:16.651525 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad03660_73d0_4233_8689_04ccbbd84555.slice/crio-451abf47b435f30082d9d34e65ec914d2bbd27c4166152bb34eb9bfdfacf8705 WatchSource:0}: Error finding container 451abf47b435f30082d9d34e65ec914d2bbd27c4166152bb34eb9bfdfacf8705: Status 404 returned error can't find the container with id 451abf47b435f30082d9d34e65ec914d2bbd27c4166152bb34eb9bfdfacf8705 Jun 10 10:50:17.146719 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:17.146679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4548f6b-xdqfc" event={"ID":"7ad03660-73d0-4233-8689-04ccbbd84555","Type":"ContainerStarted","Data":"8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a"} Jun 10 10:50:17.146922 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:17.146735 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4548f6b-xdqfc" event={"ID":"7ad03660-73d0-4233-8689-04ccbbd84555","Type":"ContainerStarted","Data":"451abf47b435f30082d9d34e65ec914d2bbd27c4166152bb34eb9bfdfacf8705"} Jun 10 10:50:17.168552 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:17.168503 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68c4548f6b-xdqfc" podStartSLOduration=1.168487498 podStartE2EDuration="1.168487498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:50:17.168375961 +0000 UTC m=+359.817431933" watchObservedRunningTime="2026-06-10 10:50:17.168487498 +0000 UTC m=+359.817543484" Jun 10 10:50:26.510524 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:26.510469 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:26.511000 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:26.510548 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:26.515404 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:26.515384 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:27.189461 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:27.189431 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:50:27.244380 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:27.244350 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f9b5c64df-bkjpk"] Jun 10 10:50:31.717981 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:31.717943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:50:31.720357 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:31.720330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8327523a-eb26-4c9d-847a-761b868b2d27-metrics-certs\") pod \"network-metrics-daemon-8t72g\" (UID: \"8327523a-eb26-4c9d-847a-761b868b2d27\") " pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:50:31.752579 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:31.752555 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgxv7\"" Jun 10 10:50:31.761025 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:31.761004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8t72g" Jun 10 10:50:31.895489 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:31.895465 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8t72g"] Jun 10 10:50:31.897970 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:50:31.897938 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8327523a_eb26_4c9d_847a_761b868b2d27.slice/crio-1393e653ba47ac4ca0960c77102a72c8e94e8440ecd37b15c5f44249e15a0e98 WatchSource:0}: Error finding container 1393e653ba47ac4ca0960c77102a72c8e94e8440ecd37b15c5f44249e15a0e98: Status 404 returned error can't find the container with id 1393e653ba47ac4ca0960c77102a72c8e94e8440ecd37b15c5f44249e15a0e98 Jun 10 10:50:31.899757 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:31.899727 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 10 10:50:32.206373 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:32.206340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8t72g" event={"ID":"8327523a-eb26-4c9d-847a-761b868b2d27","Type":"ContainerStarted","Data":"1393e653ba47ac4ca0960c77102a72c8e94e8440ecd37b15c5f44249e15a0e98"} Jun 10 10:50:33.212560 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:33.212523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8t72g" event={"ID":"8327523a-eb26-4c9d-847a-761b868b2d27","Type":"ContainerStarted","Data":"10f5112828c7eae7bcae792380b5263fcdd49a071200a39c961205c61451d689"} Jun 10 10:50:33.212991 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:33.212575 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8t72g" event={"ID":"8327523a-eb26-4c9d-847a-761b868b2d27","Type":"ContainerStarted","Data":"fd48b6b621277987ae58330eb4991c1e3f02c31e41649df52220142f90ab6649"} Jun 10 10:50:33.232236 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:33.232190 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8t72g" podStartSLOduration=374.400371438 podStartE2EDuration="6m15.232175294s" totalImagesPullingTime="831.803856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:50:33.230668659 +0000 UTC m=+375.879724632" watchObservedRunningTime="2026-06-10 10:50:33.232175294 +0000 UTC m=+375.881231266" Jun 10 10:50:52.267328 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.267267 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6f9b5c64df-bkjpk" podUID="43035f7c-dbe5-4176-8a2b-e8b6763e23a1" containerName="console" containerID="cri-o://467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85" gracePeriod=15 Jun 10 10:50:52.506560 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.506538 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:50:52.589902 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.589870 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-trusted-ca-bundle\") pod \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " Jun 10 10:50:52.589902 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.589903 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-service-ca\") pod \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " Jun 10 10:50:52.590112 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.589946 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-serving-cert\") pod \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " Jun 10 10:50:52.590112 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.589992 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-oauth-serving-cert\") pod \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " Jun 10 10:50:52.590112 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.590010 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcd4m\" (UniqueName: \"kubernetes.io/projected/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-kube-api-access-qcd4m\") pod \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " Jun 10 10:50:52.590112 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.590044 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-oauth-config\") pod \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " Jun 10 10:50:52.590112 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.590091 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-config\") pod \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\" (UID: \"43035f7c-dbe5-4176-8a2b-e8b6763e23a1\") " Jun 10 10:50:52.590364 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.590332 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "43035f7c-dbe5-4176-8a2b-e8b6763e23a1" (UID: "43035f7c-dbe5-4176-8a2b-e8b6763e23a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:50:52.590453 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.590424 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "43035f7c-dbe5-4176-8a2b-e8b6763e23a1" (UID: "43035f7c-dbe5-4176-8a2b-e8b6763e23a1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:50:52.590572 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.590469 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-service-ca" (OuterVolumeSpecName: "service-ca") pod "43035f7c-dbe5-4176-8a2b-e8b6763e23a1" (UID: "43035f7c-dbe5-4176-8a2b-e8b6763e23a1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:50:52.590572 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.590517 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-config" (OuterVolumeSpecName: "console-config") pod "43035f7c-dbe5-4176-8a2b-e8b6763e23a1" (UID: "43035f7c-dbe5-4176-8a2b-e8b6763e23a1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:50:52.592393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.592367 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-kube-api-access-qcd4m" (OuterVolumeSpecName: "kube-api-access-qcd4m") pod "43035f7c-dbe5-4176-8a2b-e8b6763e23a1" (UID: "43035f7c-dbe5-4176-8a2b-e8b6763e23a1"). InnerVolumeSpecName "kube-api-access-qcd4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:50:52.592480 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.592399 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "43035f7c-dbe5-4176-8a2b-e8b6763e23a1" (UID: "43035f7c-dbe5-4176-8a2b-e8b6763e23a1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:52.592480 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.592425 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "43035f7c-dbe5-4176-8a2b-e8b6763e23a1" (UID: "43035f7c-dbe5-4176-8a2b-e8b6763e23a1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:52.691125 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.691091 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:50:52.691125 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.691119 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-trusted-ca-bundle\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:50:52.691125 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.691129 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-service-ca\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:50:52.691339 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.691139 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:50:52.691339 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.691148 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-oauth-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:50:52.691339 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.691156 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcd4m\" (UniqueName: \"kubernetes.io/projected/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-kube-api-access-qcd4m\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:50:52.691339 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:52.691166 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43035f7c-dbe5-4176-8a2b-e8b6763e23a1-console-oauth-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:50:53.295522 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.295489 2578 generic.go:356] "Generic (PLEG): container finished" podID="43035f7c-dbe5-4176-8a2b-e8b6763e23a1" containerID="467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85" exitCode=0 Jun 10 10:50:53.296023 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.295545 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f9b5c64df-bkjpk" Jun 10 10:50:53.296023 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.295581 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f9b5c64df-bkjpk" event={"ID":"43035f7c-dbe5-4176-8a2b-e8b6763e23a1","Type":"ContainerDied","Data":"467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85"} Jun 10 10:50:53.296023 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.295645 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f9b5c64df-bkjpk" event={"ID":"43035f7c-dbe5-4176-8a2b-e8b6763e23a1","Type":"ContainerDied","Data":"76a5cc30799ee54b8e273b9150386f83a634b31c919e3a7909f767ea8bea7c44"} Jun 10 10:50:53.296023 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.295682 2578 scope.go:117] "RemoveContainer" containerID="467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85" Jun 10 10:50:53.304851 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.304835 2578 scope.go:117] "RemoveContainer" containerID="467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85" Jun 10 10:50:53.305094 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:50:53.305076 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85\": container with ID starting with 467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85 not found: ID does not exist" containerID="467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85" Jun 10 10:50:53.305129 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.305103 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85"} err="failed to get container status \"467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85\": rpc error: code = NotFound desc = could not find container \"467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85\": container with ID starting with 467a6bafbb15fa86477ae03b464531c3cd7ca3ec48694166563b1e02472cff85 not found: ID does not exist" Jun 10 10:50:53.315106 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.315083 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f9b5c64df-bkjpk"] Jun 10 10:50:53.321263 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.321241 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f9b5c64df-bkjpk"] Jun 10 10:50:53.854411 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:50:53.854367 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43035f7c-dbe5-4176-8a2b-e8b6763e23a1" path="/var/lib/kubelet/pods/43035f7c-dbe5-4176-8a2b-e8b6763e23a1/volumes" Jun 10 10:52:24.287791 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.287746 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw"] Jun 10 10:52:24.288600 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.288562 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43035f7c-dbe5-4176-8a2b-e8b6763e23a1" containerName="console" Jun 10 10:52:24.288677 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.288605 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43035f7c-dbe5-4176-8a2b-e8b6763e23a1" containerName="console" Jun 10 10:52:24.288836 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.288821 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43035f7c-dbe5-4176-8a2b-e8b6763e23a1" containerName="console" Jun 10 10:52:24.292467 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.292443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 10:52:24.295214 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.295189 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jun 10 10:52:24.295346 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.295189 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-ptzvh\"" Jun 10 10:52:24.295483 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.295464 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jun 10 10:52:24.300673 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.300652 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw"] Jun 10 10:52:24.431458 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.431424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xct\" (UniqueName: \"kubernetes.io/projected/03430f26-f2de-420e-b1b4-997940b5fccc-kube-api-access-h2xct\") pod \"feast-operator-controller-manager-69f8d7bdcc-djbkw\" (UID: \"03430f26-f2de-420e-b1b4-997940b5fccc\") " pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 10:52:24.532822 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.532755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xct\" (UniqueName: \"kubernetes.io/projected/03430f26-f2de-420e-b1b4-997940b5fccc-kube-api-access-h2xct\") pod \"feast-operator-controller-manager-69f8d7bdcc-djbkw\" (UID: \"03430f26-f2de-420e-b1b4-997940b5fccc\") " pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 10:52:24.542486 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.542428 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xct\" (UniqueName: \"kubernetes.io/projected/03430f26-f2de-420e-b1b4-997940b5fccc-kube-api-access-h2xct\") pod \"feast-operator-controller-manager-69f8d7bdcc-djbkw\" (UID: \"03430f26-f2de-420e-b1b4-997940b5fccc\") " pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 10:52:24.603975 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.603944 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 10:52:24.753774 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:24.753743 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw"] Jun 10 10:52:24.757020 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:52:24.756988 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03430f26_f2de_420e_b1b4_997940b5fccc.slice/crio-16931c4b29ed5a32440e48148396c94b3b731376168423b08562d2fc36b84d98 WatchSource:0}: Error finding container 16931c4b29ed5a32440e48148396c94b3b731376168423b08562d2fc36b84d98: Status 404 returned error can't find the container with id 16931c4b29ed5a32440e48148396c94b3b731376168423b08562d2fc36b84d98 Jun 10 10:52:25.682232 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:25.682187 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" event={"ID":"03430f26-f2de-420e-b1b4-997940b5fccc","Type":"ContainerStarted","Data":"16931c4b29ed5a32440e48148396c94b3b731376168423b08562d2fc36b84d98"} Jun 10 10:52:27.691951 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:27.691920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" event={"ID":"03430f26-f2de-420e-b1b4-997940b5fccc","Type":"ContainerStarted","Data":"5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819"} Jun 10 10:52:27.692346 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:27.692043 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 10:52:27.711422 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:27.711372 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" podStartSLOduration=1.414936582 podStartE2EDuration="3.711356098s" totalImagesPullingTime="2.296419516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:52:27.710582266 +0000 UTC m=+490.359638262" watchObservedRunningTime="2026-06-10 10:52:27.711356098 +0000 UTC m=+490.360412071" Jun 10 10:52:38.698145 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:52:38.698113 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 10:53:08.324320 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.324232 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q"] Jun 10 10:53:08.328222 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.328195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.330542 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.330515 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-9v924\"" Jun 10 10:53:08.330651 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.330610 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 10 10:53:08.331079 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.330894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 10 10:53:08.331079 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.330906 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jun 10 10:53:08.331079 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.330951 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 10 10:53:08.331079 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.330897 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 10 10:53:08.331311 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.331270 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 10 10:53:08.332462 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.332446 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 10 10:53:08.341999 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.341977 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q"] Jun 10 10:53:08.383337 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.383301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.383337 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.383337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-offline-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.383580 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.383364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-online-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.383580 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.383389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-feast-data\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.383580 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.383413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-kube-api-access-rsdqn\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.383580 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.383463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-registry-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.484302 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.484267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-online-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.484469 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.484307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-feast-data\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.484469 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.484330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-kube-api-access-rsdqn\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.484469 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.484356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-registry-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.484469 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.484439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.484469 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.484465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-offline-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.484722 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:53:08.484588 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 10 10:53:08.484722 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:53:08.484662 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls podName:c716fdd8-73de-49ba-b7c0-ffc7d4e036d2 nodeName:}" failed. No retries permitted until 2026-06-10 10:53:08.984641558 +0000 UTC m=+531.633697515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls") pod "feast-simple-feast-setup-69977fd7b4-qbb2q" (UID: "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2") : secret "feast-simple-feast-setup-ui-tls" not found Jun 10 10:53:08.484874 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.484728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-feast-data\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.487046 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.487019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-offline-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.487046 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.487021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-online-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.487207 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.487124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-registry-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.493352 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.493328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-kube-api-access-rsdqn\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.988080 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.988041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:08.990490 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:08.990467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-qbb2q\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:09.253862 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:09.253748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:09.395128 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:09.395050 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q"] Jun 10 10:53:09.397550 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:53:09.397524 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc716fdd8_73de_49ba_b7c0_ffc7d4e036d2.slice/crio-9113abe8b04f510eda0b3a25bd9397f51f2473da120b53d70dad59c7e5904b59 WatchSource:0}: Error finding container 9113abe8b04f510eda0b3a25bd9397f51f2473da120b53d70dad59c7e5904b59: Status 404 returned error can't find the container with id 9113abe8b04f510eda0b3a25bd9397f51f2473da120b53d70dad59c7e5904b59 Jun 10 10:53:09.860199 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:09.860159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerStarted","Data":"9113abe8b04f510eda0b3a25bd9397f51f2473da120b53d70dad59c7e5904b59"} Jun 10 10:53:43.018121 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:43.018081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerStarted","Data":"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e"} Jun 10 10:53:48.040609 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:48.040568 2578 generic.go:356] "Generic (PLEG): container finished" podID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerID="98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e" exitCode=0 Jun 10 10:53:48.040996 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:48.040627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerDied","Data":"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e"} Jun 10 10:53:49.047091 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:49.047049 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerStarted","Data":"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498"} Jun 10 10:53:54.069268 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:54.069231 2578 generic.go:356] "Generic (PLEG): container finished" podID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerID="b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498" exitCode=0 Jun 10 10:53:54.069646 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:54.069284 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerDied","Data":"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498"} Jun 10 10:53:54.077910 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:54.077888 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jun 10 10:53:55.080231 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:55.080183 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerStarted","Data":"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67"} Jun 10 10:53:55.080675 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:55.080247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerStarted","Data":"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265"} Jun 10 10:53:55.080675 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:55.080268 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerStarted","Data":"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02"} Jun 10 10:53:55.080675 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:55.080286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerStarted","Data":"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2"} Jun 10 10:53:55.107570 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:55.107479 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podStartSLOduration=13.701488596 podStartE2EDuration="47.107453915s" totalImagesPullingTime="33.405965319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:53:55.104825926 +0000 UTC m=+577.753881919" watchObservedRunningTime="2026-06-10 10:53:55.107453915 +0000 UTC m=+577.756509889" Jun 10 10:53:57.254306 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.254262 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:57.254778 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.254329 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:57.254778 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.254350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:57.254778 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.254367 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:57.256552 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.256417 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 10 10:53:57.256552 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.256448 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.28:8816: connect: connection refused" Jun 10 10:53:57.256552 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.256474 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 10 10:53:57.256552 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:57.256486 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.28:6571: connect: connection refused" Jun 10 10:53:59.254155 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:59.254107 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:59.254621 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:59.254183 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:59.254621 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:59.254201 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:53:59.254621 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:53:59.254221 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:00.255068 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:00.255039 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:00.255519 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:00.255407 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:00.255595 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:00.255579 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:00.259830 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:00.259788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:01.113848 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:01.113821 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:01.114027 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:01.113898 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:01.114073 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:01.114056 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:01.116251 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:01.116230 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:54:17.825094 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:17.825063 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:54:17.825640 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:17.825103 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:54:17.832159 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:17.832139 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:54:17.832343 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:17.832325 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:54:22.802833 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:22.802786 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c4548f6b-xdqfc"] Jun 10 10:54:47.822972 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:47.822904 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-68c4548f6b-xdqfc" podUID="7ad03660-73d0-4233-8689-04ccbbd84555" containerName="console" containerID="cri-o://8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a" gracePeriod=15 Jun 10 10:54:48.077062 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.076996 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:54:48.157207 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157173 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-oauth-serving-cert\") pod \"7ad03660-73d0-4233-8689-04ccbbd84555\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " Jun 10 10:54:48.157393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157215 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-trusted-ca-bundle\") pod \"7ad03660-73d0-4233-8689-04ccbbd84555\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " Jun 10 10:54:48.157393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157235 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-serving-cert\") pod \"7ad03660-73d0-4233-8689-04ccbbd84555\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " Jun 10 10:54:48.157393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157298 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-console-config\") pod \"7ad03660-73d0-4233-8689-04ccbbd84555\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " Jun 10 10:54:48.157393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157325 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-service-ca\") pod \"7ad03660-73d0-4233-8689-04ccbbd84555\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " Jun 10 10:54:48.157393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157347 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-oauth-config\") pod \"7ad03660-73d0-4233-8689-04ccbbd84555\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " Jun 10 10:54:48.157393 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157382 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l4x8\" (UniqueName: \"kubernetes.io/projected/7ad03660-73d0-4233-8689-04ccbbd84555-kube-api-access-8l4x8\") pod \"7ad03660-73d0-4233-8689-04ccbbd84555\" (UID: \"7ad03660-73d0-4233-8689-04ccbbd84555\") " Jun 10 10:54:48.157753 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157722 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7ad03660-73d0-4233-8689-04ccbbd84555" (UID: "7ad03660-73d0-4233-8689-04ccbbd84555"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:54:48.157866 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157792 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7ad03660-73d0-4233-8689-04ccbbd84555" (UID: "7ad03660-73d0-4233-8689-04ccbbd84555"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:54:48.157866 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157847 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-console-config" (OuterVolumeSpecName: "console-config") pod "7ad03660-73d0-4233-8689-04ccbbd84555" (UID: "7ad03660-73d0-4233-8689-04ccbbd84555"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:54:48.157974 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.157906 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-service-ca" (OuterVolumeSpecName: "service-ca") pod "7ad03660-73d0-4233-8689-04ccbbd84555" (UID: "7ad03660-73d0-4233-8689-04ccbbd84555"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:54:48.159647 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.159626 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7ad03660-73d0-4233-8689-04ccbbd84555" (UID: "7ad03660-73d0-4233-8689-04ccbbd84555"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:54:48.159702 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.159674 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7ad03660-73d0-4233-8689-04ccbbd84555" (UID: "7ad03660-73d0-4233-8689-04ccbbd84555"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:54:48.159702 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.159686 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad03660-73d0-4233-8689-04ccbbd84555-kube-api-access-8l4x8" (OuterVolumeSpecName: "kube-api-access-8l4x8") pod "7ad03660-73d0-4233-8689-04ccbbd84555" (UID: "7ad03660-73d0-4233-8689-04ccbbd84555"). InnerVolumeSpecName "kube-api-access-8l4x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:54:48.258982 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.258938 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-oauth-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:54:48.258982 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.258977 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-trusted-ca-bundle\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:54:48.258982 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.258987 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-serving-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:54:48.258982 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.258996 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-console-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:54:48.258982 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.259005 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad03660-73d0-4233-8689-04ccbbd84555-service-ca\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:54:48.259287 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.259012 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ad03660-73d0-4233-8689-04ccbbd84555-console-oauth-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:54:48.259287 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.259020 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4x8\" (UniqueName: \"kubernetes.io/projected/7ad03660-73d0-4233-8689-04ccbbd84555-kube-api-access-8l4x8\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:54:48.304712 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.304671 2578 generic.go:356] "Generic (PLEG): container finished" podID="7ad03660-73d0-4233-8689-04ccbbd84555" containerID="8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a" exitCode=0 Jun 10 10:54:48.304923 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.304724 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4548f6b-xdqfc" Jun 10 10:54:48.304923 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.304771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4548f6b-xdqfc" event={"ID":"7ad03660-73d0-4233-8689-04ccbbd84555","Type":"ContainerDied","Data":"8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a"} Jun 10 10:54:48.304923 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.304842 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4548f6b-xdqfc" event={"ID":"7ad03660-73d0-4233-8689-04ccbbd84555","Type":"ContainerDied","Data":"451abf47b435f30082d9d34e65ec914d2bbd27c4166152bb34eb9bfdfacf8705"} Jun 10 10:54:48.304923 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.304864 2578 scope.go:117] "RemoveContainer" containerID="8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a" Jun 10 10:54:48.314090 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.314074 2578 scope.go:117] "RemoveContainer" containerID="8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a" Jun 10 10:54:48.314336 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:54:48.314316 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a\": container with ID starting with 8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a not found: ID does not exist" containerID="8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a" Jun 10 10:54:48.314383 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.314344 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a"} err="failed to get container status \"8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a\": rpc error: code = NotFound desc = could not find container \"8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a\": container with ID starting with 8fedbd25340a698be9f698ac0c4d66db069a3df64d3c9b29335a668ebb39380a not found: ID does not exist" Jun 10 10:54:48.328159 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.328099 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c4548f6b-xdqfc"] Jun 10 10:54:48.331876 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:48.331855 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68c4548f6b-xdqfc"] Jun 10 10:54:49.854502 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:54:49.854463 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad03660-73d0-4233-8689-04ccbbd84555" path="/var/lib/kubelet/pods/7ad03660-73d0-4233-8689-04ccbbd84555/volumes" Jun 10 10:55:03.632835 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.632778 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q"] Jun 10 10:55:03.633298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.633042 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="registry" containerID="cri-o://3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2" gracePeriod=30 Jun 10 10:55:03.633298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.633157 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="offline" containerID="cri-o://5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265" gracePeriod=30 Jun 10 10:55:03.633298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.633167 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" containerID="cri-o://0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02" gracePeriod=30 Jun 10 10:55:03.633464 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.633370 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" containerID="cri-o://769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67" gracePeriod=30 Jun 10 10:55:03.898723 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.898638 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf"] Jun 10 10:55:03.899236 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.899213 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad03660-73d0-4233-8689-04ccbbd84555" containerName="console" Jun 10 10:55:03.899407 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.899394 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad03660-73d0-4233-8689-04ccbbd84555" containerName="console" Jun 10 10:55:03.899603 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.899590 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad03660-73d0-4233-8689-04ccbbd84555" containerName="console" Jun 10 10:55:03.907858 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.906684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:03.910921 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.908987 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-dsj4p\"" Jun 10 10:55:03.919790 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.919317 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf"] Jun 10 10:55:03.988410 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.988367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:03.988592 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.988421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-online-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:03.988592 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.988505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-feast-data\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:03.988592 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.988547 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvhf\" (UniqueName: \"kubernetes.io/projected/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-kube-api-access-njvhf\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:03.988722 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.988594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-registry-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:03.988722 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:03.988690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-offline-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.089452 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.089413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-online-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.089602 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.089465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-feast-data\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.089602 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.089490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvhf\" (UniqueName: \"kubernetes.io/projected/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-kube-api-access-njvhf\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.089602 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.089533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-registry-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.089602 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.089583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-offline-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.089837 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.089628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.089837 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:04.089742 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 10 10:55:04.089837 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:04.089821 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls podName:6d13d4ca-b9c7-4a89-a413-2a751f60d9b8 nodeName:}" failed. No retries permitted until 2026-06-10 10:55:04.589781882 +0000 UTC m=+647.238837843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls") pod "feast-simple-feast-setup-69977fd7b4-z4vmf" (UID: "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8") : secret "feast-simple-feast-setup-ui-tls" not found Jun 10 10:55:04.090652 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.090623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-feast-data\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.092458 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.092435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-online-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.093050 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.093023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-offline-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.093156 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.093136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-registry-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.100070 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.100046 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvhf\" (UniqueName: \"kubernetes.io/projected/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-kube-api-access-njvhf\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.371543 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.371515 2578 generic.go:356] "Generic (PLEG): container finished" podID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerID="769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67" exitCode=0 Jun 10 10:55:04.371543 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.371539 2578 generic.go:356] "Generic (PLEG): container finished" podID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerID="0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02" exitCode=0 Jun 10 10:55:04.371704 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.371585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerDied","Data":"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67"} Jun 10 10:55:04.371704 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.371627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerDied","Data":"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02"} Jun 10 10:55:04.594208 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.594177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.596659 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.596641 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls\") pod \"feast-simple-feast-setup-69977fd7b4-z4vmf\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.822550 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.822511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:04.960717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:04.960686 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf"] Jun 10 10:55:04.963689 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:55:04.963665 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d13d4ca_b9c7_4a89_a413_2a751f60d9b8.slice/crio-f236bd92e91412b036f698702162ef81cead196e12c6296f16890cf00a0cef6c WatchSource:0}: Error finding container f236bd92e91412b036f698702162ef81cead196e12c6296f16890cf00a0cef6c: Status 404 returned error can't find the container with id f236bd92e91412b036f698702162ef81cead196e12c6296f16890cf00a0cef6c Jun 10 10:55:05.378673 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:05.378638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerStarted","Data":"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6"} Jun 10 10:55:05.378883 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:05.378685 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerStarted","Data":"f236bd92e91412b036f698702162ef81cead196e12c6296f16890cf00a0cef6c"} Jun 10 10:55:10.402810 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:10.402755 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerID="2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6" exitCode=0 Jun 10 10:55:10.403205 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:10.402829 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerDied","Data":"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6"} Jun 10 10:55:11.114119 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:11.114065 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 10 10:55:11.114325 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:11.114085 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 10 10:55:11.409145 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:11.409059 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerStarted","Data":"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502"} Jun 10 10:55:16.431962 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:16.431929 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerID="d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502" exitCode=0 Jun 10 10:55:16.432329 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:16.432008 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerDied","Data":"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502"} Jun 10 10:55:17.442373 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:17.442325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerStarted","Data":"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290"} Jun 10 10:55:17.442846 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:17.442391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerStarted","Data":"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7"} Jun 10 10:55:17.442846 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:17.442417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerStarted","Data":"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1"} Jun 10 10:55:17.442846 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:17.442436 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerStarted","Data":"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496"} Jun 10 10:55:17.469296 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:17.469066 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podStartSLOduration=14.469043379 podStartE2EDuration="14.469043379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:55:17.467514703 +0000 UTC m=+660.116570688" watchObservedRunningTime="2026-06-10 10:55:17.469043379 +0000 UTC m=+660.118099353" Jun 10 10:55:19.823132 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.823068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:19.823132 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.823142 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:19.823662 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.823162 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:19.823662 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.823179 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:19.825439 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.825396 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused" Jun 10 10:55:19.825780 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.825750 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jun 10 10:55:19.826015 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.825753 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jun 10 10:55:19.826128 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:19.825790 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused" Jun 10 10:55:21.114220 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:21.113857 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 10 10:55:21.114784 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:21.114454 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 10 10:55:22.824117 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.824087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.824594 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.824281 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.824594 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.824453 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.825026 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.825000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.825159 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.825044 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.825159 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.825065 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.825265 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.825206 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.825390 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.825364 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.825636 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.825620 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:22.829135 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:22.829118 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:23.477415 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:23.477379 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:23.480187 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:23.480168 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:55:31.114207 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:31.114140 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.28:8443: connect: connection refused" Jun 10 10:55:31.114207 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:31.114146 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.28:6567/health\": dial tcp 10.132.0.28:6567: connect: connection refused" Jun 10 10:55:31.114663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:31.114343 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:55:31.114663 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:31.114508 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:55:34.282786 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.282762 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:55:34.349152 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.349125 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-kube-api-access-rsdqn\") pod \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " Jun 10 10:55:34.349313 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.349164 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-feast-data\") pod \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " Jun 10 10:55:34.349313 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.349189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-registry-tls\") pod \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " Jun 10 10:55:34.349313 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.349221 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-offline-tls\") pod \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " Jun 10 10:55:34.349313 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.349278 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-online-tls\") pod \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " Jun 10 10:55:34.349478 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.349320 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls\") pod \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\" (UID: \"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2\") " Jun 10 10:55:34.349734 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.349701 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-feast-data" (OuterVolumeSpecName: "feast-data") pod "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" (UID: "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 10:55:34.351753 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.351726 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" (UID: "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:55:34.351892 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.351748 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" (UID: "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:55:34.351892 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.351771 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-kube-api-access-rsdqn" (OuterVolumeSpecName: "kube-api-access-rsdqn") pod "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" (UID: "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2"). InnerVolumeSpecName "kube-api-access-rsdqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:55:34.352083 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.352066 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" (UID: "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:55:34.352125 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.352081 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-online-tls" (OuterVolumeSpecName: "online-tls") pod "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" (UID: "c716fdd8-73de-49ba-b7c0-ffc7d4e036d2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:55:34.450722 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.450644 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-ui-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:55:34.450722 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.450670 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsdqn\" (UniqueName: \"kubernetes.io/projected/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-kube-api-access-rsdqn\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:55:34.450722 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.450682 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-feast-data\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:55:34.450722 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.450691 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:55:34.450722 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.450701 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-offline-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:55:34.451009 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.450733 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2-online-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:55:34.523726 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.523685 2578 generic.go:356] "Generic (PLEG): container finished" podID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerID="5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265" exitCode=137 Jun 10 10:55:34.523726 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.523709 2578 generic.go:356] "Generic (PLEG): container finished" podID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerID="3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2" exitCode=137 Jun 10 10:55:34.523909 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.523765 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" Jun 10 10:55:34.523909 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.523765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerDied","Data":"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265"} Jun 10 10:55:34.523909 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.523844 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerDied","Data":"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2"} Jun 10 10:55:34.523909 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.523865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q" event={"ID":"c716fdd8-73de-49ba-b7c0-ffc7d4e036d2","Type":"ContainerDied","Data":"9113abe8b04f510eda0b3a25bd9397f51f2473da120b53d70dad59c7e5904b59"} Jun 10 10:55:34.523909 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.523889 2578 scope.go:117] "RemoveContainer" containerID="769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67" Jun 10 10:55:34.533454 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.533432 2578 scope.go:117] "RemoveContainer" containerID="5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265" Jun 10 10:55:34.541083 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.541067 2578 scope.go:117] "RemoveContainer" containerID="0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02" Jun 10 10:55:34.547754 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.547729 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q"] Jun 10 10:55:34.548971 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.548947 2578 scope.go:117] "RemoveContainer" containerID="3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2" Jun 10 10:55:34.551313 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.551294 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-qbb2q"] Jun 10 10:55:34.556737 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.556720 2578 scope.go:117] "RemoveContainer" containerID="b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498" Jun 10 10:55:34.564906 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.564888 2578 scope.go:117] "RemoveContainer" containerID="98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e" Jun 10 10:55:34.572275 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.572243 2578 scope.go:117] "RemoveContainer" containerID="769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67" Jun 10 10:55:34.572565 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:34.572544 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67\": container with ID starting with 769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67 not found: ID does not exist" containerID="769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67" Jun 10 10:55:34.572643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.572573 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67"} err="failed to get container status \"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67\": rpc error: code = NotFound desc = could not find container \"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67\": container with ID starting with 769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67 not found: ID does not exist" Jun 10 10:55:34.572643 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.572591 2578 scope.go:117] "RemoveContainer" containerID="5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265" Jun 10 10:55:34.572854 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:34.572833 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265\": container with ID starting with 5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265 not found: ID does not exist" containerID="5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265" Jun 10 10:55:34.572913 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.572864 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265"} err="failed to get container status \"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265\": rpc error: code = NotFound desc = could not find container \"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265\": container with ID starting with 5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265 not found: ID does not exist" Jun 10 10:55:34.572913 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.572887 2578 scope.go:117] "RemoveContainer" containerID="0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02" Jun 10 10:55:34.573145 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:34.573128 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02\": container with ID starting with 0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02 not found: ID does not exist" containerID="0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02" Jun 10 10:55:34.573194 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573150 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02"} err="failed to get container status \"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02\": rpc error: code = NotFound desc = could not find container \"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02\": container with ID starting with 0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02 not found: ID does not exist" Jun 10 10:55:34.573194 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573162 2578 scope.go:117] "RemoveContainer" containerID="3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2" Jun 10 10:55:34.573368 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:34.573353 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2\": container with ID starting with 3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2 not found: ID does not exist" containerID="3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2" Jun 10 10:55:34.573414 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573370 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2"} err="failed to get container status \"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2\": rpc error: code = NotFound desc = could not find container \"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2\": container with ID starting with 3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2 not found: ID does not exist" Jun 10 10:55:34.573414 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573383 2578 scope.go:117] "RemoveContainer" containerID="b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498" Jun 10 10:55:34.573568 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:34.573550 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498\": container with ID starting with b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498 not found: ID does not exist" containerID="b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498" Jun 10 10:55:34.573644 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573571 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498"} err="failed to get container status \"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498\": rpc error: code = NotFound desc = could not find container \"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498\": container with ID starting with b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498 not found: ID does not exist" Jun 10 10:55:34.573644 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573582 2578 scope.go:117] "RemoveContainer" containerID="98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e" Jun 10 10:55:34.573777 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:55:34.573764 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e\": container with ID starting with 98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e not found: ID does not exist" containerID="98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e" Jun 10 10:55:34.573859 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573780 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e"} err="failed to get container status \"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e\": rpc error: code = NotFound desc = could not find container \"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e\": container with ID starting with 98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e not found: ID does not exist" Jun 10 10:55:34.573859 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.573792 2578 scope.go:117] "RemoveContainer" containerID="769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67" Jun 10 10:55:34.574043 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574021 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67"} err="failed to get container status \"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67\": rpc error: code = NotFound desc = could not find container \"769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67\": container with ID starting with 769a2d79495f5d93976c5a0ed84b4fe01a9387bed6747448cdec2bde2b8c3d67 not found: ID does not exist" Jun 10 10:55:34.574110 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574046 2578 scope.go:117] "RemoveContainer" containerID="5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265" Jun 10 10:55:34.574333 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574311 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265"} err="failed to get container status \"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265\": rpc error: code = NotFound desc = could not find container \"5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265\": container with ID starting with 5feb5d2905d572843c88683967075da0b9820f9a6f2ef038fff020cc7d9d9265 not found: ID does not exist" Jun 10 10:55:34.574385 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574335 2578 scope.go:117] "RemoveContainer" containerID="0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02" Jun 10 10:55:34.574575 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574554 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02"} err="failed to get container status \"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02\": rpc error: code = NotFound desc = could not find container \"0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02\": container with ID starting with 0ecf740dadeeeb3a7ca55dc14bf4fddc910a3819254735b08069a7c653b99e02 not found: ID does not exist" Jun 10 10:55:34.574575 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574575 2578 scope.go:117] "RemoveContainer" containerID="3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2" Jun 10 10:55:34.574788 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574765 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2"} err="failed to get container status \"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2\": rpc error: code = NotFound desc = could not find container \"3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2\": container with ID starting with 3016db313cccbfd360419ee49efde3927e6ce6c14fca708b76083d1b728275a2 not found: ID does not exist" Jun 10 10:55:34.574788 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.574785 2578 scope.go:117] "RemoveContainer" containerID="b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498" Jun 10 10:55:34.575018 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.575003 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498"} err="failed to get container status \"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498\": rpc error: code = NotFound desc = could not find container \"b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498\": container with ID starting with b27a2fe8520ab1df3f6ed2e1edb355e1d545f3119e5d207d6611c9ccd520a498 not found: ID does not exist" Jun 10 10:55:34.575018 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.575018 2578 scope.go:117] "RemoveContainer" containerID="98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e" Jun 10 10:55:34.575231 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:34.575211 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e"} err="failed to get container status \"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e\": rpc error: code = NotFound desc = could not find container \"98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e\": container with ID starting with 98685d3dd02518e82a823630e891c3ad88c3953ad2ebee29f1f5759b90f3b23e not found: ID does not exist" Jun 10 10:55:35.854065 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:55:35.854032 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" path="/var/lib/kubelet/pods/c716fdd8-73de-49ba-b7c0-ffc7d4e036d2/volumes" Jun 10 10:56:25.593785 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.593749 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p"] Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594123 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="feast-init" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594137 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="feast-init" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594155 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594161 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594170 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="registry" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594180 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="registry" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594191 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="feast-apply" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594196 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="feast-apply" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594202 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594207 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594217 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="offline" Jun 10 10:56:25.594230 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594221 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="offline" Jun 10 10:56:25.594578 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594282 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="offline" Jun 10 10:56:25.594578 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594291 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="online" Jun 10 10:56:25.594578 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594297 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="ui" Jun 10 10:56:25.594578 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.594307 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c716fdd8-73de-49ba-b7c0-ffc7d4e036d2" containerName="registry" Jun 10 10:56:25.599171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.599149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.602298 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602267 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 10 10:56:25.602481 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602306 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 10 10:56:25.602760 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602348 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 10 10:56:25.602907 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602369 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 10 10:56:25.602995 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602385 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 10 10:56:25.603077 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602410 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 10 10:56:25.603142 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602470 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-vlrx4\"" Jun 10 10:56:25.603264 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.602469 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jun 10 10:56:25.612212 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.612187 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p"] Jun 10 10:56:25.666971 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.666930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4a199457-d205-46d3-933c-6d588cdfe1bc-feast-data\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.666971 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.666989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.667247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.667013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4a199457-d205-46d3-933c-6d588cdfe1bc-registry-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.667247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.667087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-online-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.667247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.667146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.667247 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.667170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v674m\" (UniqueName: \"kubernetes.io/projected/4a199457-d205-46d3-933c-6d588cdfe1bc-kube-api-access-v674m\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768084 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4a199457-d205-46d3-933c-6d588cdfe1bc-feast-data\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768276 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768276 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4a199457-d205-46d3-933c-6d588cdfe1bc-registry-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768276 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-online-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768276 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768276 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v674m\" (UniqueName: \"kubernetes.io/projected/4a199457-d205-46d3-933c-6d588cdfe1bc-kube-api-access-v674m\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768540 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:56:25.768305 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 10 10:56:25.768540 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:56:25.768383 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls podName:4a199457-d205-46d3-933c-6d588cdfe1bc nodeName:}" failed. No retries permitted until 2026-06-10 10:56:26.268359413 +0000 UTC m=+728.917415379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls") pod "feast-simple-feast-remote-setup-5f869d56c7-m8n2p" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 10 10:56:25.768540 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768527 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4a199457-d205-46d3-933c-6d588cdfe1bc-feast-data\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.768696 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:56:25.768545 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 10 10:56:25.768696 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:56:25.768613 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls podName:4a199457-d205-46d3-933c-6d588cdfe1bc nodeName:}" failed. No retries permitted until 2026-06-10 10:56:26.268596059 +0000 UTC m=+728.917652012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls") pod "feast-simple-feast-remote-setup-5f869d56c7-m8n2p" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 10 10:56:25.768935 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.768915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4a199457-d205-46d3-933c-6d588cdfe1bc-registry-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.770812 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.770772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-online-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:25.779459 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:25.779434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v674m\" (UniqueName: \"kubernetes.io/projected/4a199457-d205-46d3-933c-6d588cdfe1bc-kube-api-access-v674m\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:26.272428 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.272385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:26.272603 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.272444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:26.275025 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.275000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:26.275125 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.275050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls\") pod \"feast-simple-feast-remote-setup-5f869d56c7-m8n2p\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:26.510717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.510674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:26.650163 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.650128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p"] Jun 10 10:56:26.653183 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:56:26.653156 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a199457_d205_46d3_933c_6d588cdfe1bc.slice/crio-6346ec9175546ce6eef3b00b379f0f3a39bf566b279a67857c2b22983e7a06ba WatchSource:0}: Error finding container 6346ec9175546ce6eef3b00b379f0f3a39bf566b279a67857c2b22983e7a06ba: Status 404 returned error can't find the container with id 6346ec9175546ce6eef3b00b379f0f3a39bf566b279a67857c2b22983e7a06ba Jun 10 10:56:26.747705 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.747661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerStarted","Data":"9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab"} Jun 10 10:56:26.747891 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:26.747733 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerStarted","Data":"6346ec9175546ce6eef3b00b379f0f3a39bf566b279a67857c2b22983e7a06ba"} Jun 10 10:56:31.769025 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:31.768990 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerID="9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab" exitCode=0 Jun 10 10:56:31.769426 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:31.769046 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerDied","Data":"9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab"} Jun 10 10:56:32.774771 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:32.774735 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerStarted","Data":"dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6"} Jun 10 10:56:37.799592 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:37.799507 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerID="dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6" exitCode=0 Jun 10 10:56:37.799592 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:37.799581 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerDied","Data":"dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6"} Jun 10 10:56:38.809356 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:38.809318 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerStarted","Data":"fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492"} Jun 10 10:56:38.809863 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:38.809382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerStarted","Data":"91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341"} Jun 10 10:56:38.809863 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:38.809403 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerStarted","Data":"f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b"} Jun 10 10:56:38.833524 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:38.833463 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podStartSLOduration=13.833444792 podStartE2EDuration="13.833444792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:56:38.8314121 +0000 UTC m=+741.480468101" watchObservedRunningTime="2026-06-10 10:56:38.833444792 +0000 UTC m=+741.482500765" Jun 10 10:56:41.511679 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:41.511634 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:41.512191 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:41.511703 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:41.512191 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:41.511729 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:41.513367 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:41.513324 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" Jun 10 10:56:41.513367 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:41.513343 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jun 10 10:56:41.513535 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:41.513383 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 10 10:56:44.512365 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.512326 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.512917 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.512853 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.512917 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.512894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.513134 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.513115 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.516653 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.516631 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.839520 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.839483 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.839520 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.839528 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.840136 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.840116 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:56:44.842398 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:56:44.842374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:57:47.333614 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.333573 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p"] Jun 10 10:57:47.334200 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.333898 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" containerID="cri-o://f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b" gracePeriod=30 Jun 10 10:57:47.334200 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.333917 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="offline" containerID="cri-o://91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341" gracePeriod=30 Jun 10 10:57:47.334200 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.333960 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" containerID="cri-o://fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492" gracePeriod=30 Jun 10 10:57:47.447050 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.447018 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf"] Jun 10 10:57:47.447294 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.447261 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="registry" containerID="cri-o://ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496" gracePeriod=30 Jun 10 10:57:47.447392 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.447300 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" containerID="cri-o://a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290" gracePeriod=30 Jun 10 10:57:47.447392 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.447311 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" containerID="cri-o://609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1" gracePeriod=30 Jun 10 10:57:47.447392 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:47.447337 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="offline" containerID="cri-o://1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7" gracePeriod=30 Jun 10 10:57:48.099489 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:48.099453 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerID="fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492" exitCode=0 Jun 10 10:57:48.099489 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:48.099481 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerID="f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b" exitCode=0 Jun 10 10:57:48.099748 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:48.099523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerDied","Data":"fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492"} Jun 10 10:57:48.099748 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:48.099564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerDied","Data":"f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b"} Jun 10 10:57:48.102616 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:48.102591 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerID="a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290" exitCode=0 Jun 10 10:57:48.102766 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:48.102664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerDied","Data":"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290"} Jun 10 10:57:49.109949 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:49.109916 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerID="609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1" exitCode=0 Jun 10 10:57:49.110305 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:49.109991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerDied","Data":"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1"} Jun 10 10:57:52.825172 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:52.825109 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jun 10 10:57:53.478010 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:53.477970 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jun 10 10:57:54.840152 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:54.840104 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jun 10 10:57:54.840537 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:57:54.840126 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 10 10:58:02.825369 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:02.825330 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jun 10 10:58:03.478115 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:03.478071 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jun 10 10:58:04.839636 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:04.839588 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jun 10 10:58:04.840030 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:04.839632 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 10 10:58:12.825131 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:12.825069 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused" Jun 10 10:58:12.825594 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:12.825206 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:58:13.477717 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:13.477674 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused" Jun 10 10:58:13.477931 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:13.477834 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:58:14.840471 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:14.840431 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused" Jun 10 10:58:14.840868 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:14.840442 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" Jun 10 10:58:14.840868 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:14.840538 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:58:14.840868 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:14.840586 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:58:17.582913 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.582892 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:58:17.585992 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.585971 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls\") pod \"4a199457-d205-46d3-933c-6d588cdfe1bc\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " Jun 10 10:58:17.586061 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.586010 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4a199457-d205-46d3-933c-6d588cdfe1bc-registry-tls\") pod \"4a199457-d205-46d3-933c-6d588cdfe1bc\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " Jun 10 10:58:17.586061 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.586029 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls\") pod \"4a199457-d205-46d3-933c-6d588cdfe1bc\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " Jun 10 10:58:17.586061 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.586054 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-online-tls\") pod \"4a199457-d205-46d3-933c-6d588cdfe1bc\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " Jun 10 10:58:17.586218 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.586084 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4a199457-d205-46d3-933c-6d588cdfe1bc-feast-data\") pod \"4a199457-d205-46d3-933c-6d588cdfe1bc\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " Jun 10 10:58:17.586218 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.586116 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v674m\" (UniqueName: \"kubernetes.io/projected/4a199457-d205-46d3-933c-6d588cdfe1bc-kube-api-access-v674m\") pod \"4a199457-d205-46d3-933c-6d588cdfe1bc\" (UID: \"4a199457-d205-46d3-933c-6d588cdfe1bc\") " Jun 10 10:58:17.586396 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.586369 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a199457-d205-46d3-933c-6d588cdfe1bc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4a199457-d205-46d3-933c-6d588cdfe1bc" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:58:17.586606 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.586583 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a199457-d205-46d3-933c-6d588cdfe1bc-feast-data" (OuterVolumeSpecName: "feast-data") pod "4a199457-d205-46d3-933c-6d588cdfe1bc" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 10:58:17.588435 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.588361 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4a199457-d205-46d3-933c-6d588cdfe1bc" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:58:17.588541 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.588518 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a199457-d205-46d3-933c-6d588cdfe1bc-kube-api-access-v674m" (OuterVolumeSpecName: "kube-api-access-v674m") pod "4a199457-d205-46d3-933c-6d588cdfe1bc" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc"). InnerVolumeSpecName "kube-api-access-v674m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:58:17.588691 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.588668 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4a199457-d205-46d3-933c-6d588cdfe1bc" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:58:17.588733 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.588699 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-online-tls" (OuterVolumeSpecName: "online-tls") pod "4a199457-d205-46d3-933c-6d588cdfe1bc" (UID: "4a199457-d205-46d3-933c-6d588cdfe1bc"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:58:17.686675 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.686640 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v674m\" (UniqueName: \"kubernetes.io/projected/4a199457-d205-46d3-933c-6d588cdfe1bc-kube-api-access-v674m\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:17.686675 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.686668 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-ui-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:17.686675 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.686681 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4a199457-d205-46d3-933c-6d588cdfe1bc-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:17.686925 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.686692 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-offline-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:17.686925 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.686703 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4a199457-d205-46d3-933c-6d588cdfe1bc-online-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:17.686925 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:17.686713 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4a199457-d205-46d3-933c-6d588cdfe1bc-feast-data\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:18.089420 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.089394 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:58:18.189150 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.189059 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-offline-tls\") pod \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " Jun 10 10:58:18.189150 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.189103 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njvhf\" (UniqueName: \"kubernetes.io/projected/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-kube-api-access-njvhf\") pod \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " Jun 10 10:58:18.189150 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.189125 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls\") pod \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " Jun 10 10:58:18.189398 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.189159 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-feast-data\") pod \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " Jun 10 10:58:18.189398 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.189177 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-online-tls\") pod \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " Jun 10 10:58:18.189398 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.189199 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-registry-tls\") pod \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\" (UID: \"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8\") " Jun 10 10:58:18.189758 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.189718 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-feast-data" (OuterVolumeSpecName: "feast-data") pod "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" (UID: "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 10:58:18.191495 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.191459 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-online-tls" (OuterVolumeSpecName: "online-tls") pod "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" (UID: "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:58:18.191581 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.191546 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" (UID: "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:58:18.191581 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.191558 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" (UID: "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:58:18.191883 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.191857 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" (UID: "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:58:18.191883 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.191866 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-kube-api-access-njvhf" (OuterVolumeSpecName: "kube-api-access-njvhf") pod "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" (UID: "6d13d4ca-b9c7-4a89-a413-2a751f60d9b8"). InnerVolumeSpecName "kube-api-access-njvhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:58:18.229334 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.229299 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerID="91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341" exitCode=137 Jun 10 10:58:18.229461 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.229379 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" Jun 10 10:58:18.229461 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.229384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerDied","Data":"91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341"} Jun 10 10:58:18.229461 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.229435 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p" event={"ID":"4a199457-d205-46d3-933c-6d588cdfe1bc","Type":"ContainerDied","Data":"6346ec9175546ce6eef3b00b379f0f3a39bf566b279a67857c2b22983e7a06ba"} Jun 10 10:58:18.229624 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.229462 2578 scope.go:117] "RemoveContainer" containerID="fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492" Jun 10 10:58:18.232517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.232495 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerID="1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7" exitCode=137 Jun 10 10:58:18.232517 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.232514 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerID="ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496" exitCode=137 Jun 10 10:58:18.232660 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.232573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerDied","Data":"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7"} Jun 10 10:58:18.232660 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.232615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerDied","Data":"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496"} Jun 10 10:58:18.232660 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.232637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" event={"ID":"6d13d4ca-b9c7-4a89-a413-2a751f60d9b8","Type":"ContainerDied","Data":"f236bd92e91412b036f698702162ef81cead196e12c6296f16890cf00a0cef6c"} Jun 10 10:58:18.232660 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.232647 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf" Jun 10 10:58:18.239455 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.239437 2578 scope.go:117] "RemoveContainer" containerID="91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341" Jun 10 10:58:18.247426 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.247411 2578 scope.go:117] "RemoveContainer" containerID="f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b" Jun 10 10:58:18.254588 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.254572 2578 scope.go:117] "RemoveContainer" containerID="dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6" Jun 10 10:58:18.261147 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.261120 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p"] Jun 10 10:58:18.262480 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.262464 2578 scope.go:117] "RemoveContainer" containerID="9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab" Jun 10 10:58:18.270358 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.270333 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5f869d56c7-m8n2p"] Jun 10 10:58:18.276825 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.276780 2578 scope.go:117] "RemoveContainer" containerID="fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492" Jun 10 10:58:18.277117 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.277097 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492\": container with ID starting with fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492 not found: ID does not exist" containerID="fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492" Jun 10 10:58:18.277171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277124 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492"} err="failed to get container status \"fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492\": rpc error: code = NotFound desc = could not find container \"fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492\": container with ID starting with fdd676fd485802d2e3cd86dae48d6b3f18236db26c33b71630e936a8d4857492 not found: ID does not exist" Jun 10 10:58:18.277171 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277140 2578 scope.go:117] "RemoveContainer" containerID="91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341" Jun 10 10:58:18.277358 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.277340 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341\": container with ID starting with 91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341 not found: ID does not exist" containerID="91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341" Jun 10 10:58:18.277402 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277363 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341"} err="failed to get container status \"91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341\": rpc error: code = NotFound desc = could not find container \"91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341\": container with ID starting with 91f4b64d9472b996db003a227ad9d1420595f118b87753e235f232400a49f341 not found: ID does not exist" Jun 10 10:58:18.277402 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277376 2578 scope.go:117] "RemoveContainer" containerID="f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b" Jun 10 10:58:18.277602 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.277584 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b\": container with ID starting with f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b not found: ID does not exist" containerID="f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b" Jun 10 10:58:18.277669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277621 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b"} err="failed to get container status \"f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b\": rpc error: code = NotFound desc = could not find container \"f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b\": container with ID starting with f3e4ba7db8fac0ca15058fa8c5d2ef3f6e7b163233b205454ebd1f2db468a28b not found: ID does not exist" Jun 10 10:58:18.277669 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277643 2578 scope.go:117] "RemoveContainer" containerID="dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6" Jun 10 10:58:18.277902 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.277886 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6\": container with ID starting with dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6 not found: ID does not exist" containerID="dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6" Jun 10 10:58:18.277967 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277906 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6"} err="failed to get container status \"dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6\": rpc error: code = NotFound desc = could not find container \"dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6\": container with ID starting with dfcf184b1bab98e2e4eb90fa9c5f735b0e891965795a4f818bc3857e20aa8bc6 not found: ID does not exist" Jun 10 10:58:18.277967 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.277919 2578 scope.go:117] "RemoveContainer" containerID="9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab" Jun 10 10:58:18.278235 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.278210 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab\": container with ID starting with 9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab not found: ID does not exist" containerID="9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab" Jun 10 10:58:18.278296 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.278234 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab"} err="failed to get container status \"9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab\": rpc error: code = NotFound desc = could not find container \"9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab\": container with ID starting with 9d936d773147346a40132d1c651eefc347ecdc9b1fc8b6d6ad9f6c5e1ca9a2ab not found: ID does not exist" Jun 10 10:58:18.278296 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.278253 2578 scope.go:117] "RemoveContainer" containerID="a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290" Jun 10 10:58:18.286229 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.286213 2578 scope.go:117] "RemoveContainer" containerID="1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7" Jun 10 10:58:18.289759 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.289740 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-offline-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:18.289850 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.289766 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njvhf\" (UniqueName: \"kubernetes.io/projected/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-kube-api-access-njvhf\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:18.289850 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.289780 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-ui-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:18.289850 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.289810 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-feast-data\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:18.289850 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.289824 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-online-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:18.289850 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.289835 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 10:58:18.294204 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.294162 2578 scope.go:117] "RemoveContainer" containerID="609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1" Jun 10 10:58:18.295637 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.295618 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf"] Jun 10 10:58:18.300640 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.300619 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-69977fd7b4-z4vmf"] Jun 10 10:58:18.303652 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.303637 2578 scope.go:117] "RemoveContainer" containerID="ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496" Jun 10 10:58:18.311456 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.311434 2578 scope.go:117] "RemoveContainer" containerID="d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502" Jun 10 10:58:18.318513 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.318497 2578 scope.go:117] "RemoveContainer" containerID="2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6" Jun 10 10:58:18.325518 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.325501 2578 scope.go:117] "RemoveContainer" containerID="a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290" Jun 10 10:58:18.325746 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.325728 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290\": container with ID starting with a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290 not found: ID does not exist" containerID="a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290" Jun 10 10:58:18.325820 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.325754 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290"} err="failed to get container status \"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290\": rpc error: code = NotFound desc = could not find container \"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290\": container with ID starting with a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290 not found: ID does not exist" Jun 10 10:58:18.325820 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.325772 2578 scope.go:117] "RemoveContainer" containerID="1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7" Jun 10 10:58:18.326022 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.326003 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7\": container with ID starting with 1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7 not found: ID does not exist" containerID="1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7" Jun 10 10:58:18.326060 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326028 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7"} err="failed to get container status \"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7\": rpc error: code = NotFound desc = could not find container \"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7\": container with ID starting with 1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7 not found: ID does not exist" Jun 10 10:58:18.326060 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326043 2578 scope.go:117] "RemoveContainer" containerID="609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1" Jun 10 10:58:18.326279 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.326264 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1\": container with ID starting with 609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1 not found: ID does not exist" containerID="609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1" Jun 10 10:58:18.326322 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326281 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1"} err="failed to get container status \"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1\": rpc error: code = NotFound desc = could not find container \"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1\": container with ID starting with 609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1 not found: ID does not exist" Jun 10 10:58:18.326322 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326292 2578 scope.go:117] "RemoveContainer" containerID="ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496" Jun 10 10:58:18.326476 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.326461 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496\": container with ID starting with ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496 not found: ID does not exist" containerID="ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496" Jun 10 10:58:18.326510 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326479 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496"} err="failed to get container status \"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496\": rpc error: code = NotFound desc = could not find container \"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496\": container with ID starting with ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496 not found: ID does not exist" Jun 10 10:58:18.326510 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326490 2578 scope.go:117] "RemoveContainer" containerID="d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502" Jun 10 10:58:18.326707 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.326692 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502\": container with ID starting with d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502 not found: ID does not exist" containerID="d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502" Jun 10 10:58:18.326761 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326709 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502"} err="failed to get container status \"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502\": rpc error: code = NotFound desc = could not find container \"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502\": container with ID starting with d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502 not found: ID does not exist" Jun 10 10:58:18.326761 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326721 2578 scope.go:117] "RemoveContainer" containerID="2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6" Jun 10 10:58:18.326951 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:18.326935 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6\": container with ID starting with 2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6 not found: ID does not exist" containerID="2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6" Jun 10 10:58:18.326988 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326953 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6"} err="failed to get container status \"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6\": rpc error: code = NotFound desc = could not find container \"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6\": container with ID starting with 2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6 not found: ID does not exist" Jun 10 10:58:18.326988 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.326964 2578 scope.go:117] "RemoveContainer" containerID="a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290" Jun 10 10:58:18.327160 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327138 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290"} err="failed to get container status \"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290\": rpc error: code = NotFound desc = could not find container \"a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290\": container with ID starting with a4eb02ca56888c25a490dcc458d03cac5db7fdf28d519e78fcb062c8c8d2c290 not found: ID does not exist" Jun 10 10:58:18.327206 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327161 2578 scope.go:117] "RemoveContainer" containerID="1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7" Jun 10 10:58:18.327374 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327359 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7"} err="failed to get container status \"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7\": rpc error: code = NotFound desc = could not find container \"1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7\": container with ID starting with 1c04d6574cccf2a59140bfef0fa6d1db879b0c5de3c8ac0064ff6db727d2fcc7 not found: ID does not exist" Jun 10 10:58:18.327418 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327373 2578 scope.go:117] "RemoveContainer" containerID="609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1" Jun 10 10:58:18.327579 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327561 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1"} err="failed to get container status \"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1\": rpc error: code = NotFound desc = could not find container \"609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1\": container with ID starting with 609ef46c84545d467e1d52f70e8405bac920427cbf25d4faad3c54e3ddf609a1 not found: ID does not exist" Jun 10 10:58:18.327641 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327582 2578 scope.go:117] "RemoveContainer" containerID="ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496" Jun 10 10:58:18.327853 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327833 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496"} err="failed to get container status \"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496\": rpc error: code = NotFound desc = could not find container \"ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496\": container with ID starting with ba8a57f0d60c915081787d564b4f90e96be4bc7fbbc73a379734bd4da883c496 not found: ID does not exist" Jun 10 10:58:18.327908 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.327857 2578 scope.go:117] "RemoveContainer" containerID="d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502" Jun 10 10:58:18.328054 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.328038 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502"} err="failed to get container status \"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502\": rpc error: code = NotFound desc = could not find container \"d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502\": container with ID starting with d1c95d88c10a9afa35f3ce909094f52d4af23da8e9aed23bc83f5f9cd2b04502 not found: ID does not exist" Jun 10 10:58:18.328114 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.328054 2578 scope.go:117] "RemoveContainer" containerID="2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6" Jun 10 10:58:18.328218 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:18.328202 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6"} err="failed to get container status \"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6\": rpc error: code = NotFound desc = could not find container \"2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6\": container with ID starting with 2de7c68c364b8d5bc10a08e459af5f5e8b7a954517f9d3446d50ee9081ff00d6 not found: ID does not exist" Jun 10 10:58:19.854215 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:19.854181 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" path="/var/lib/kubelet/pods/4a199457-d205-46d3-933c-6d588cdfe1bc/volumes" Jun 10 10:58:19.854710 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:19.854697 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" path="/var/lib/kubelet/pods/6d13d4ca-b9c7-4a89-a413-2a751f60d9b8/volumes" Jun 10 10:58:30.324486 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324452 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6jfk"] Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324773 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="offline" Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324785 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="offline" Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324816 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="feast-init" Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324824 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="feast-init" Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324836 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324842 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324848 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" Jun 10 10:58:30.324852 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324853 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324862 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="feast-init" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324867 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="feast-init" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324873 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="feast-apply" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324880 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="feast-apply" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324894 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324899 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324906 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="offline" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324910 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="offline" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324916 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="feast-apply" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324920 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="feast-apply" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324926 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="registry" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324930 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="registry" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324938 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324943 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.324994 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="online" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.325003 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="online" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.325011 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="offline" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.325016 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="ui" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.325022 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a199457-d205-46d3-933c-6d588cdfe1bc" containerName="offline" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.325029 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="registry" Jun 10 10:58:30.325097 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.325035 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d13d4ca-b9c7-4a89-a413-2a751f60d9b8" containerName="ui" Jun 10 10:58:30.330041 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.330024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.333011 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.332985 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 10 10:58:30.333124 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.333014 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jun 10 10:58:30.333124 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.333052 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-lp54l\"" Jun 10 10:58:30.333632 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.333613 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 10 10:58:30.337590 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.337571 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6jfk"] Jun 10 10:58:30.357074 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.357036 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-dwsgv"] Jun 10 10:58:30.362035 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.362005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-dwsgv" Jun 10 10:58:30.370695 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.370672 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-dwsgv"] Jun 10 10:58:30.374963 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.374938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhbj\" (UniqueName: \"kubernetes.io/projected/3ede196a-f341-4dbb-866e-f054a51d14db-kube-api-access-mvhbj\") pod \"postgres-864846d4f5-w6jfk\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.375083 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.375008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ede196a-f341-4dbb-866e-f054a51d14db-postgresdata\") pod \"postgres-864846d4f5-w6jfk\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.375083 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.375052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5v8\" (UniqueName: \"kubernetes.io/projected/9d375c44-4c40-4bcd-8b96-27c55c7c9a33-kube-api-access-9g5v8\") pod \"redis-69bbd798b6-dwsgv\" (UID: \"9d375c44-4c40-4bcd-8b96-27c55c7c9a33\") " pod="test-ns-feast/redis-69bbd798b6-dwsgv" Jun 10 10:58:30.475937 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.475902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ede196a-f341-4dbb-866e-f054a51d14db-postgresdata\") pod \"postgres-864846d4f5-w6jfk\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.475937 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.475942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5v8\" (UniqueName: \"kubernetes.io/projected/9d375c44-4c40-4bcd-8b96-27c55c7c9a33-kube-api-access-9g5v8\") pod \"redis-69bbd798b6-dwsgv\" (UID: \"9d375c44-4c40-4bcd-8b96-27c55c7c9a33\") " pod="test-ns-feast/redis-69bbd798b6-dwsgv" Jun 10 10:58:30.476155 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.475993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhbj\" (UniqueName: \"kubernetes.io/projected/3ede196a-f341-4dbb-866e-f054a51d14db-kube-api-access-mvhbj\") pod \"postgres-864846d4f5-w6jfk\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.476262 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.476241 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ede196a-f341-4dbb-866e-f054a51d14db-postgresdata\") pod \"postgres-864846d4f5-w6jfk\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.485205 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.485179 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5v8\" (UniqueName: \"kubernetes.io/projected/9d375c44-4c40-4bcd-8b96-27c55c7c9a33-kube-api-access-9g5v8\") pod \"redis-69bbd798b6-dwsgv\" (UID: \"9d375c44-4c40-4bcd-8b96-27c55c7c9a33\") " pod="test-ns-feast/redis-69bbd798b6-dwsgv" Jun 10 10:58:30.485316 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.485294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhbj\" (UniqueName: \"kubernetes.io/projected/3ede196a-f341-4dbb-866e-f054a51d14db-kube-api-access-mvhbj\") pod \"postgres-864846d4f5-w6jfk\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.655951 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.655861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 10:58:30.671086 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.671062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-dwsgv" Jun 10 10:58:30.799585 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.799558 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6jfk"] Jun 10 10:58:30.802337 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:58:30.802310 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ede196a_f341_4dbb_866e_f054a51d14db.slice/crio-c5897a1faa90539812e40ade7b8f64a16eef3f6bb19f05259fb18859152a7f0a WatchSource:0}: Error finding container c5897a1faa90539812e40ade7b8f64a16eef3f6bb19f05259fb18859152a7f0a: Status 404 returned error can't find the container with id c5897a1faa90539812e40ade7b8f64a16eef3f6bb19f05259fb18859152a7f0a Jun 10 10:58:30.803932 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.803916 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 10 10:58:30.819048 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:30.819026 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-dwsgv"] Jun 10 10:58:30.821272 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:58:30.821242 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d375c44_4c40_4bcd_8b96_27c55c7c9a33.slice/crio-6058474da2eca5d5050ab142c945b14a54818ce7348d8aaa0c68ec2381f50e4b WatchSource:0}: Error finding container 6058474da2eca5d5050ab142c945b14a54818ce7348d8aaa0c68ec2381f50e4b: Status 404 returned error can't find the container with id 6058474da2eca5d5050ab142c945b14a54818ce7348d8aaa0c68ec2381f50e4b Jun 10 10:58:31.285431 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:31.285393 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-dwsgv" event={"ID":"9d375c44-4c40-4bcd-8b96-27c55c7c9a33","Type":"ContainerStarted","Data":"6058474da2eca5d5050ab142c945b14a54818ce7348d8aaa0c68ec2381f50e4b"} Jun 10 10:58:31.286386 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:31.286364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6jfk" event={"ID":"3ede196a-f341-4dbb-866e-f054a51d14db","Type":"ContainerStarted","Data":"c5897a1faa90539812e40ade7b8f64a16eef3f6bb19f05259fb18859152a7f0a"} Jun 10 10:58:33.296648 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:33.296612 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-dwsgv" event={"ID":"9d375c44-4c40-4bcd-8b96-27c55c7c9a33","Type":"ContainerStarted","Data":"12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9"} Jun 10 10:58:33.317413 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:33.317362 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-dwsgv" podStartSLOduration=1.8010687239999998 podStartE2EDuration="3.317282135s" totalImagesPullingTime="1.516213411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:58:33.315403943 +0000 UTC m=+855.964459917" watchObservedRunningTime="2026-06-10 10:58:33.317282135 +0000 UTC m=+855.966338107" Jun 10 10:58:36.315213 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:36.315176 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6jfk" event={"ID":"3ede196a-f341-4dbb-866e-f054a51d14db","Type":"ContainerStarted","Data":"be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172"} Jun 10 10:58:36.337223 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:36.337181 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-w6jfk" podStartSLOduration=1.190027988 podStartE2EDuration="6.337167855s" totalImagesPullingTime="5.147139867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:58:36.335748423 +0000 UTC m=+858.984804395" watchObservedRunningTime="2026-06-10 10:58:36.337167855 +0000 UTC m=+858.986223827" Jun 10 10:58:39.131510 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.131472 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-5df794694d-t4gxt"] Jun 10 10:58:39.135587 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.135569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.139331 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.139308 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-zsxr2\"" Jun 10 10:58:39.140061 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.140039 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jun 10 10:58:39.140210 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.140191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jun 10 10:58:39.140768 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.140746 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jun 10 10:58:39.141032 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.141016 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jun 10 10:58:39.152582 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.152559 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5df794694d-t4gxt"] Jun 10 10:58:39.259080 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.259050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-online-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.259250 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.259087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.259250 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.259117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-offline-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.259250 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.259176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4g4\" (UniqueName: \"kubernetes.io/projected/9038782a-d9c7-48ae-9d6c-24290303d10b-kube-api-access-vg4g4\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.259250 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.259238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9038782a-d9c7-48ae-9d6c-24290303d10b-feast-data\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.360523 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.360488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9038782a-d9c7-48ae-9d6c-24290303d10b-feast-data\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.360713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.360546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-online-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.360713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.360565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.360713 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:39.360645 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jun 10 10:58:39.360713 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.360677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-offline-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.360713 ip-10-0-138-231 kubenswrapper[2578]: E0610 10:58:39.360696 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls podName:9038782a-d9c7-48ae-9d6c-24290303d10b nodeName:}" failed. No retries permitted until 2026-06-10 10:58:39.860681222 +0000 UTC m=+862.509737173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls") pod "feast-credit-scoring-5df794694d-t4gxt" (UID: "9038782a-d9c7-48ae-9d6c-24290303d10b") : secret "feast-credit-scoring-registry-tls" not found Jun 10 10:58:39.361022 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.360728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4g4\" (UniqueName: \"kubernetes.io/projected/9038782a-d9c7-48ae-9d6c-24290303d10b-kube-api-access-vg4g4\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.361022 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.360959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9038782a-d9c7-48ae-9d6c-24290303d10b-feast-data\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.363332 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.363302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-offline-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.363444 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.363311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-online-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.374305 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.374283 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4g4\" (UniqueName: \"kubernetes.io/projected/9038782a-d9c7-48ae-9d6c-24290303d10b-kube-api-access-vg4g4\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.865407 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.865380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:39.868041 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:39.868013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls\") pod \"feast-credit-scoring-5df794694d-t4gxt\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:40.045877 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:40.045846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:40.195605 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:40.195582 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5df794694d-t4gxt"] Jun 10 10:58:40.197570 ip-10-0-138-231 kubenswrapper[2578]: W0610 10:58:40.197546 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9038782a_d9c7_48ae_9d6c_24290303d10b.slice/crio-4cf933d0d9284acb573e1717d04e59b97a4cdb5330c5d90d9c72b2b049270766 WatchSource:0}: Error finding container 4cf933d0d9284acb573e1717d04e59b97a4cdb5330c5d90d9c72b2b049270766: Status 404 returned error can't find the container with id 4cf933d0d9284acb573e1717d04e59b97a4cdb5330c5d90d9c72b2b049270766 Jun 10 10:58:40.333284 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:40.333248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerStarted","Data":"c70fdf91144246b9225982059778a3ea34f8ba7ce14b206ebadd64879d7781d6"} Jun 10 10:58:40.333478 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:40.333314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerStarted","Data":"4cf933d0d9284acb573e1717d04e59b97a4cdb5330c5d90d9c72b2b049270766"} Jun 10 10:58:41.339030 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:41.338992 2578 generic.go:356] "Generic (PLEG): container finished" podID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerID="c70fdf91144246b9225982059778a3ea34f8ba7ce14b206ebadd64879d7781d6" exitCode=0 Jun 10 10:58:41.339374 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:41.339073 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerDied","Data":"c70fdf91144246b9225982059778a3ea34f8ba7ce14b206ebadd64879d7781d6"} Jun 10 10:58:42.345387 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:42.345346 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerStarted","Data":"4924c271ca4e32229cafdfb85cdff51e674e0231188e1932a4a7a2744c578152"} Jun 10 10:58:48.370444 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:48.370408 2578 generic.go:356] "Generic (PLEG): container finished" podID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerID="4924c271ca4e32229cafdfb85cdff51e674e0231188e1932a4a7a2744c578152" exitCode=0 Jun 10 10:58:48.370877 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:48.370452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerDied","Data":"4924c271ca4e32229cafdfb85cdff51e674e0231188e1932a4a7a2744c578152"} Jun 10 10:58:49.379708 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:49.379671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerStarted","Data":"0dc5c95ca70130340a3e6846a75849e5ffee92be782b813a74ce36338913de95"} Jun 10 10:58:49.380135 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:49.379749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerStarted","Data":"70f90e663c4dd178a8a27bb0d2e022d08e364997c227d592ebe151bfba1b0c03"} Jun 10 10:58:49.380135 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:49.379779 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerStarted","Data":"5008425e2811da6ab29c6e03bb05bc5e82b01a9be194befa11474788e8d4f8f0"} Jun 10 10:58:49.408859 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:49.408750 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podStartSLOduration=10.408731946 podStartE2EDuration="10.408731946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:58:49.408331576 +0000 UTC m=+872.057387549" watchObservedRunningTime="2026-06-10 10:58:49.408731946 +0000 UTC m=+872.057787958" Jun 10 10:58:50.046478 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:50.046443 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:50.046639 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:50.046532 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:50.046639 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:50.046559 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:52.046748 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:52.046701 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:52.047242 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:52.046785 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:52.047242 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:52.046831 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:52.048871 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:52.048839 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.33:8816: connect: connection refused" Jun 10 10:58:52.049110 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:52.048875 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 10 10:58:52.049222 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:52.048873 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Jun 10 10:58:55.046543 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:55.046492 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Jun 10 10:58:55.047202 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:55.047172 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:55.048202 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:55.048178 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:55.052140 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:55.052120 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:55.416986 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:55.416959 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:58:58.046879 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:58:58.046833 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" Jun 10 10:59:01.047509 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:59:01.047476 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:59:01.048220 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:59:01.048203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 10:59:17.855362 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:59:17.855285 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:59:17.857446 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:59:17.857424 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 10:59:17.863214 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:59:17.863195 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 10:59:17.864856 ip-10-0-138-231 kubenswrapper[2578]: I0610 10:59:17.864840 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:00:02.884289 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:02.884252 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5df794694d-t4gxt"] Jun 10 11:00:02.892636 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:02.892611 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6jfk"] Jun 10 11:00:02.892823 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:02.892788 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-w6jfk" podUID="3ede196a-f341-4dbb-866e-f054a51d14db" containerName="postgres" containerID="cri-o://be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172" gracePeriod=30 Jun 10 11:00:02.897165 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:02.897141 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-dwsgv"] Jun 10 11:00:02.897313 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:02.897282 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-dwsgv" podUID="9d375c44-4c40-4bcd-8b96-27c55c7c9a33" containerName="redis" containerID="cri-o://12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9" gracePeriod=30 Jun 10 11:00:03.170944 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.170922 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 11:00:03.174009 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.173991 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-dwsgv" Jun 10 11:00:03.287070 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.287039 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvhbj\" (UniqueName: \"kubernetes.io/projected/3ede196a-f341-4dbb-866e-f054a51d14db-kube-api-access-mvhbj\") pod \"3ede196a-f341-4dbb-866e-f054a51d14db\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " Jun 10 11:00:03.287070 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.287077 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ede196a-f341-4dbb-866e-f054a51d14db-postgresdata\") pod \"3ede196a-f341-4dbb-866e-f054a51d14db\" (UID: \"3ede196a-f341-4dbb-866e-f054a51d14db\") " Jun 10 11:00:03.287325 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.287095 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g5v8\" (UniqueName: \"kubernetes.io/projected/9d375c44-4c40-4bcd-8b96-27c55c7c9a33-kube-api-access-9g5v8\") pod \"9d375c44-4c40-4bcd-8b96-27c55c7c9a33\" (UID: \"9d375c44-4c40-4bcd-8b96-27c55c7c9a33\") " Jun 10 11:00:03.289303 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.289267 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ede196a-f341-4dbb-866e-f054a51d14db-kube-api-access-mvhbj" (OuterVolumeSpecName: "kube-api-access-mvhbj") pod "3ede196a-f341-4dbb-866e-f054a51d14db" (UID: "3ede196a-f341-4dbb-866e-f054a51d14db"). InnerVolumeSpecName "kube-api-access-mvhbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:00:03.289442 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.289384 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d375c44-4c40-4bcd-8b96-27c55c7c9a33-kube-api-access-9g5v8" (OuterVolumeSpecName: "kube-api-access-9g5v8") pod "9d375c44-4c40-4bcd-8b96-27c55c7c9a33" (UID: "9d375c44-4c40-4bcd-8b96-27c55c7c9a33"). InnerVolumeSpecName "kube-api-access-9g5v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:00:03.316395 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.316365 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ede196a-f341-4dbb-866e-f054a51d14db-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "3ede196a-f341-4dbb-866e-f054a51d14db" (UID: "3ede196a-f341-4dbb-866e-f054a51d14db"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:00:03.387931 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.387898 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvhbj\" (UniqueName: \"kubernetes.io/projected/3ede196a-f341-4dbb-866e-f054a51d14db-kube-api-access-mvhbj\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:03.387931 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.387925 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ede196a-f341-4dbb-866e-f054a51d14db-postgresdata\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:03.388087 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.387938 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5v8\" (UniqueName: \"kubernetes.io/projected/9d375c44-4c40-4bcd-8b96-27c55c7c9a33-kube-api-access-9g5v8\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:03.710076 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.710039 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d375c44-4c40-4bcd-8b96-27c55c7c9a33" containerID="12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9" exitCode=0 Jun 10 11:00:03.710266 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.710081 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-dwsgv" Jun 10 11:00:03.710266 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.710126 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-dwsgv" event={"ID":"9d375c44-4c40-4bcd-8b96-27c55c7c9a33","Type":"ContainerDied","Data":"12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9"} Jun 10 11:00:03.710266 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.710162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-dwsgv" event={"ID":"9d375c44-4c40-4bcd-8b96-27c55c7c9a33","Type":"ContainerDied","Data":"6058474da2eca5d5050ab142c945b14a54818ce7348d8aaa0c68ec2381f50e4b"} Jun 10 11:00:03.710266 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.710176 2578 scope.go:117] "RemoveContainer" containerID="12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9" Jun 10 11:00:03.711588 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.711567 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-w6jfk" Jun 10 11:00:03.711696 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.711652 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6jfk" event={"ID":"3ede196a-f341-4dbb-866e-f054a51d14db","Type":"ContainerDied","Data":"be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172"} Jun 10 11:00:03.711815 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.711771 2578 generic.go:356] "Generic (PLEG): container finished" podID="3ede196a-f341-4dbb-866e-f054a51d14db" containerID="be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172" exitCode=0 Jun 10 11:00:03.712012 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.711986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-w6jfk" event={"ID":"3ede196a-f341-4dbb-866e-f054a51d14db","Type":"ContainerDied","Data":"c5897a1faa90539812e40ade7b8f64a16eef3f6bb19f05259fb18859152a7f0a"} Jun 10 11:00:03.712205 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.712179 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="registry" containerID="cri-o://5008425e2811da6ab29c6e03bb05bc5e82b01a9be194befa11474788e8d4f8f0" gracePeriod=30 Jun 10 11:00:03.712278 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.712220 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="offline" containerID="cri-o://0dc5c95ca70130340a3e6846a75849e5ffee92be782b813a74ce36338913de95" gracePeriod=30 Jun 10 11:00:03.712278 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.712226 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" containerID="cri-o://70f90e663c4dd178a8a27bb0d2e022d08e364997c227d592ebe151bfba1b0c03" gracePeriod=30 Jun 10 11:00:03.720303 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.720271 2578 scope.go:117] "RemoveContainer" containerID="12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9" Jun 10 11:00:03.720588 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:00:03.720561 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9\": container with ID starting with 12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9 not found: ID does not exist" containerID="12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9" Jun 10 11:00:03.720668 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.720600 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9"} err="failed to get container status \"12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9\": rpc error: code = NotFound desc = could not find container \"12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9\": container with ID starting with 12638d1c73a5d4e3dee80547c08ff0a9dc697c0766b106812280550c5dbb71b9 not found: ID does not exist" Jun 10 11:00:03.720668 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.720624 2578 scope.go:117] "RemoveContainer" containerID="be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172" Jun 10 11:00:03.728191 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.728175 2578 scope.go:117] "RemoveContainer" containerID="be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172" Jun 10 11:00:03.728409 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:00:03.728391 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172\": container with ID starting with be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172 not found: ID does not exist" containerID="be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172" Jun 10 11:00:03.728451 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.728418 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172"} err="failed to get container status \"be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172\": rpc error: code = NotFound desc = could not find container \"be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172\": container with ID starting with be0a2140623de64119840bb60758192355bbdba3d95317907996eb99c635b172 not found: ID does not exist" Jun 10 11:00:03.801647 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.801608 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6jfk"] Jun 10 11:00:03.824028 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.823998 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-w6jfk"] Jun 10 11:00:03.854109 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.854080 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ede196a-f341-4dbb-866e-f054a51d14db" path="/var/lib/kubelet/pods/3ede196a-f341-4dbb-866e-f054a51d14db/volumes" Jun 10 11:00:03.856265 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.856245 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-dwsgv"] Jun 10 11:00:03.868887 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:03.868858 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-dwsgv"] Jun 10 11:00:04.721149 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:04.721062 2578 generic.go:356] "Generic (PLEG): container finished" podID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerID="70f90e663c4dd178a8a27bb0d2e022d08e364997c227d592ebe151bfba1b0c03" exitCode=0 Jun 10 11:00:04.721149 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:04.721099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerDied","Data":"70f90e663c4dd178a8a27bb0d2e022d08e364997c227d592ebe151bfba1b0c03"} Jun 10 11:00:05.414464 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:05.414419 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 10 11:00:05.854941 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:05.854907 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d375c44-4c40-4bcd-8b96-27c55c7c9a33" path="/var/lib/kubelet/pods/9d375c44-4c40-4bcd-8b96-27c55c7c9a33/volumes" Jun 10 11:00:15.414627 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:15.414576 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 10 11:00:25.414033 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:25.413981 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" Jun 10 11:00:25.414412 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:25.414158 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 11:00:33.843230 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:33.843189 2578 generic.go:356] "Generic (PLEG): container finished" podID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerID="0dc5c95ca70130340a3e6846a75849e5ffee92be782b813a74ce36338913de95" exitCode=137 Jun 10 11:00:33.843230 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:33.843215 2578 generic.go:356] "Generic (PLEG): container finished" podID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerID="5008425e2811da6ab29c6e03bb05bc5e82b01a9be194befa11474788e8d4f8f0" exitCode=137 Jun 10 11:00:33.843664 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:33.843260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerDied","Data":"0dc5c95ca70130340a3e6846a75849e5ffee92be782b813a74ce36338913de95"} Jun 10 11:00:33.843664 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:33.843341 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerDied","Data":"5008425e2811da6ab29c6e03bb05bc5e82b01a9be194befa11474788e8d4f8f0"} Jun 10 11:00:34.361059 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.361031 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 11:00:34.421008 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.420971 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls\") pod \"9038782a-d9c7-48ae-9d6c-24290303d10b\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " Jun 10 11:00:34.421008 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.421008 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9038782a-d9c7-48ae-9d6c-24290303d10b-feast-data\") pod \"9038782a-d9c7-48ae-9d6c-24290303d10b\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " Jun 10 11:00:34.421256 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.421046 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg4g4\" (UniqueName: \"kubernetes.io/projected/9038782a-d9c7-48ae-9d6c-24290303d10b-kube-api-access-vg4g4\") pod \"9038782a-d9c7-48ae-9d6c-24290303d10b\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " Jun 10 11:00:34.421256 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.421083 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-offline-tls\") pod \"9038782a-d9c7-48ae-9d6c-24290303d10b\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " Jun 10 11:00:34.421256 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.421144 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-online-tls\") pod \"9038782a-d9c7-48ae-9d6c-24290303d10b\" (UID: \"9038782a-d9c7-48ae-9d6c-24290303d10b\") " Jun 10 11:00:34.423471 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.423428 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9038782a-d9c7-48ae-9d6c-24290303d10b" (UID: "9038782a-d9c7-48ae-9d6c-24290303d10b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:00:34.423604 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.423509 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-online-tls" (OuterVolumeSpecName: "online-tls") pod "9038782a-d9c7-48ae-9d6c-24290303d10b" (UID: "9038782a-d9c7-48ae-9d6c-24290303d10b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:00:34.423604 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.423514 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9038782a-d9c7-48ae-9d6c-24290303d10b-kube-api-access-vg4g4" (OuterVolumeSpecName: "kube-api-access-vg4g4") pod "9038782a-d9c7-48ae-9d6c-24290303d10b" (UID: "9038782a-d9c7-48ae-9d6c-24290303d10b"). InnerVolumeSpecName "kube-api-access-vg4g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:00:34.423604 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.423565 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9038782a-d9c7-48ae-9d6c-24290303d10b" (UID: "9038782a-d9c7-48ae-9d6c-24290303d10b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:00:34.429851 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.429815 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9038782a-d9c7-48ae-9d6c-24290303d10b-feast-data" (OuterVolumeSpecName: "feast-data") pod "9038782a-d9c7-48ae-9d6c-24290303d10b" (UID: "9038782a-d9c7-48ae-9d6c-24290303d10b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:00:34.522213 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.522132 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:34.522213 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.522160 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9038782a-d9c7-48ae-9d6c-24290303d10b-feast-data\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:34.522213 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.522172 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4g4\" (UniqueName: \"kubernetes.io/projected/9038782a-d9c7-48ae-9d6c-24290303d10b-kube-api-access-vg4g4\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:34.522213 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.522180 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-offline-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:34.522213 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.522188 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9038782a-d9c7-48ae-9d6c-24290303d10b-online-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:00:34.849645 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.849622 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" Jun 10 11:00:34.850060 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.849622 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5df794694d-t4gxt" event={"ID":"9038782a-d9c7-48ae-9d6c-24290303d10b","Type":"ContainerDied","Data":"4cf933d0d9284acb573e1717d04e59b97a4cdb5330c5d90d9c72b2b049270766"} Jun 10 11:00:34.850060 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.849753 2578 scope.go:117] "RemoveContainer" containerID="0dc5c95ca70130340a3e6846a75849e5ffee92be782b813a74ce36338913de95" Jun 10 11:00:34.859733 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.859707 2578 scope.go:117] "RemoveContainer" containerID="70f90e663c4dd178a8a27bb0d2e022d08e364997c227d592ebe151bfba1b0c03" Jun 10 11:00:34.868492 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.868469 2578 scope.go:117] "RemoveContainer" containerID="5008425e2811da6ab29c6e03bb05bc5e82b01a9be194befa11474788e8d4f8f0" Jun 10 11:00:34.873745 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.873718 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5df794694d-t4gxt"] Jun 10 11:00:34.877404 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.877382 2578 scope.go:117] "RemoveContainer" containerID="4924c271ca4e32229cafdfb85cdff51e674e0231188e1932a4a7a2744c578152" Jun 10 11:00:34.879851 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.879824 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-5df794694d-t4gxt"] Jun 10 11:00:34.885915 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:34.885896 2578 scope.go:117] "RemoveContainer" containerID="c70fdf91144246b9225982059778a3ea34f8ba7ce14b206ebadd64879d7781d6" Jun 10 11:00:35.853579 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:00:35.853547 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" path="/var/lib/kubelet/pods/9038782a-d9c7-48ae-9d6c-24290303d10b/volumes" Jun 10 11:02:15.503567 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503489 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4w4v7"] Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503828 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d375c44-4c40-4bcd-8b96-27c55c7c9a33" containerName="redis" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503841 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d375c44-4c40-4bcd-8b96-27c55c7c9a33" containerName="redis" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503850 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="registry" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503856 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="registry" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503863 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ede196a-f341-4dbb-866e-f054a51d14db" containerName="postgres" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503868 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ede196a-f341-4dbb-866e-f054a51d14db" containerName="postgres" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503878 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="feast-apply" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503883 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="feast-apply" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503891 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="offline" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503896 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="offline" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503903 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503909 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503923 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="feast-init" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503929 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="feast-init" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503979 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="offline" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503988 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="online" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.503995 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d375c44-4c40-4bcd-8b96-27c55c7c9a33" containerName="redis" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.504004 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ede196a-f341-4dbb-866e-f054a51d14db" containerName="postgres" Jun 10 11:02:15.504061 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.504008 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9038782a-d9c7-48ae-9d6c-24290303d10b" containerName="registry" Jun 10 11:02:15.506845 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.506827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" Jun 10 11:02:15.510380 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.510352 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jun 10 11:02:15.510839 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.510820 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-722xs\"" Jun 10 11:02:15.511234 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.511220 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jun 10 11:02:15.528239 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.528212 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4w4v7"] Jun 10 11:02:15.663787 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.663743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9j8k\" (UniqueName: \"kubernetes.io/projected/538e9571-c302-4a94-8278-49a67076dbed-kube-api-access-w9j8k\") pod \"redis-69bbd798b6-4w4v7\" (UID: \"538e9571-c302-4a94-8278-49a67076dbed\") " pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" Jun 10 11:02:15.765133 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.765040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9j8k\" (UniqueName: \"kubernetes.io/projected/538e9571-c302-4a94-8278-49a67076dbed-kube-api-access-w9j8k\") pod \"redis-69bbd798b6-4w4v7\" (UID: \"538e9571-c302-4a94-8278-49a67076dbed\") " pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" Jun 10 11:02:15.777064 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.777035 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9j8k\" (UniqueName: \"kubernetes.io/projected/538e9571-c302-4a94-8278-49a67076dbed-kube-api-access-w9j8k\") pod \"redis-69bbd798b6-4w4v7\" (UID: \"538e9571-c302-4a94-8278-49a67076dbed\") " pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" Jun 10 11:02:15.823880 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.823840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" Jun 10 11:02:15.966362 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:15.966326 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4w4v7"] Jun 10 11:02:15.969766 ip-10-0-138-231 kubenswrapper[2578]: W0610 11:02:15.969726 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod538e9571_c302_4a94_8278_49a67076dbed.slice/crio-e23f2181be93e475a2d4d24c28eb2bc614dc1dcdb165d6af59c5bc94e25052c4 WatchSource:0}: Error finding container e23f2181be93e475a2d4d24c28eb2bc614dc1dcdb165d6af59c5bc94e25052c4: Status 404 returned error can't find the container with id e23f2181be93e475a2d4d24c28eb2bc614dc1dcdb165d6af59c5bc94e25052c4 Jun 10 11:02:16.255780 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:16.255746 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" event={"ID":"538e9571-c302-4a94-8278-49a67076dbed","Type":"ContainerStarted","Data":"99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c"} Jun 10 11:02:16.255974 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:16.255791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" event={"ID":"538e9571-c302-4a94-8278-49a67076dbed","Type":"ContainerStarted","Data":"e23f2181be93e475a2d4d24c28eb2bc614dc1dcdb165d6af59c5bc94e25052c4"} Jun 10 11:02:16.276940 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:16.276789 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" podStartSLOduration=1.2767687620000001 podStartE2EDuration="1.276768762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:02:16.275678114 +0000 UTC m=+1078.924734087" watchObservedRunningTime="2026-06-10 11:02:16.276768762 +0000 UTC m=+1078.925824736" Jun 10 11:02:20.978221 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:20.978185 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-txcrh"] Jun 10 11:02:20.981280 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:20.981261 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:20.983481 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:20.983460 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jun 10 11:02:20.992593 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:20.992573 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-txcrh"] Jun 10 11:02:21.004039 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.004017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-postgresdata\") pod \"postgres-864846d4f5-txcrh\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:21.004140 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.004076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6kq\" (UniqueName: \"kubernetes.io/projected/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-kube-api-access-zt6kq\") pod \"postgres-864846d4f5-txcrh\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:21.104763 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.104726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6kq\" (UniqueName: \"kubernetes.io/projected/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-kube-api-access-zt6kq\") pod \"postgres-864846d4f5-txcrh\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:21.104964 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.104767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-postgresdata\") pod \"postgres-864846d4f5-txcrh\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:21.105147 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.105126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-postgresdata\") pod \"postgres-864846d4f5-txcrh\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:21.117264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.117234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6kq\" (UniqueName: \"kubernetes.io/projected/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-kube-api-access-zt6kq\") pod \"postgres-864846d4f5-txcrh\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:21.290850 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.290755 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:02:21.425815 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:21.425745 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-txcrh"] Jun 10 11:02:21.433005 ip-10-0-138-231 kubenswrapper[2578]: W0610 11:02:21.432962 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf34b5d0_2b91_4a6a_918a_401d45f8cd2e.slice/crio-c06e8e6718fa072ede307d12f5c9982c1ae9ede7b6fd6938e53de5fcd6270c16 WatchSource:0}: Error finding container c06e8e6718fa072ede307d12f5c9982c1ae9ede7b6fd6938e53de5fcd6270c16: Status 404 returned error can't find the container with id c06e8e6718fa072ede307d12f5c9982c1ae9ede7b6fd6938e53de5fcd6270c16 Jun 10 11:02:22.282443 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:22.282413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" event={"ID":"af34b5d0-2b91-4a6a-918a-401d45f8cd2e","Type":"ContainerStarted","Data":"0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4"} Jun 10 11:02:22.282443 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:22.282457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" event={"ID":"af34b5d0-2b91-4a6a-918a-401d45f8cd2e","Type":"ContainerStarted","Data":"c06e8e6718fa072ede307d12f5c9982c1ae9ede7b6fd6938e53de5fcd6270c16"} Jun 10 11:02:22.301495 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:22.301451 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" podStartSLOduration=2.301434672 podStartE2EDuration="2.301434672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:02:22.300990978 +0000 UTC m=+1084.950046981" watchObservedRunningTime="2026-06-10 11:02:22.301434672 +0000 UTC m=+1084.950490644" Jun 10 11:02:26.728310 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.728272 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994"] Jun 10 11:02:26.731011 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.730992 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.734292 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.734267 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jun 10 11:02:26.734493 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.734474 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jun 10 11:02:26.734583 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.734551 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jun 10 11:02:26.734583 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.734548 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-jxcln\"" Jun 10 11:02:26.742080 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.742060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64b580db-adec-498b-a97f-26340d7f6b0d-feast-data\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.742189 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.742087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-registry-tls\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.742189 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.742136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-online-tls\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.742189 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.742158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ttg\" (UniqueName: \"kubernetes.io/projected/64b580db-adec-498b-a97f-26340d7f6b0d-kube-api-access-64ttg\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.750937 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.750906 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994"] Jun 10 11:02:26.843136 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.843097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64b580db-adec-498b-a97f-26340d7f6b0d-feast-data\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.843335 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.843140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-registry-tls\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.843335 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.843209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-online-tls\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.843335 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.843284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ttg\" (UniqueName: \"kubernetes.io/projected/64b580db-adec-498b-a97f-26340d7f6b0d-kube-api-access-64ttg\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.843543 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.843523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64b580db-adec-498b-a97f-26340d7f6b0d-feast-data\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.845858 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.845836 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-online-tls\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.845971 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.845898 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-registry-tls\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:26.852417 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:26.852392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ttg\" (UniqueName: \"kubernetes.io/projected/64b580db-adec-498b-a97f-26340d7f6b0d-kube-api-access-64ttg\") pod \"feast-credit-scoring-6944f8477c-79994\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:27.041605 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:27.041510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:27.189321 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:27.189293 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994"] Jun 10 11:02:27.192215 ip-10-0-138-231 kubenswrapper[2578]: W0610 11:02:27.192184 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b580db_adec_498b_a97f_26340d7f6b0d.slice/crio-1e5e32b68ce8cb92bdf94f95c68027dd8f6b43087a9da9bef115b2fa08d7a739 WatchSource:0}: Error finding container 1e5e32b68ce8cb92bdf94f95c68027dd8f6b43087a9da9bef115b2fa08d7a739: Status 404 returned error can't find the container with id 1e5e32b68ce8cb92bdf94f95c68027dd8f6b43087a9da9bef115b2fa08d7a739 Jun 10 11:02:27.304914 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:27.304793 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerStarted","Data":"b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87"} Jun 10 11:02:27.304914 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:27.304869 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerStarted","Data":"1e5e32b68ce8cb92bdf94f95c68027dd8f6b43087a9da9bef115b2fa08d7a739"} Jun 10 11:02:28.310838 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:28.310781 2578 generic.go:356] "Generic (PLEG): container finished" podID="64b580db-adec-498b-a97f-26340d7f6b0d" containerID="b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87" exitCode=0 Jun 10 11:02:28.311235 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:28.310859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerDied","Data":"b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87"} Jun 10 11:02:29.317866 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:29.317827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerStarted","Data":"0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194"} Jun 10 11:02:35.347861 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:35.347827 2578 generic.go:356] "Generic (PLEG): container finished" podID="64b580db-adec-498b-a97f-26340d7f6b0d" containerID="0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194" exitCode=0 Jun 10 11:02:35.348189 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:35.347903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerDied","Data":"0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194"} Jun 10 11:02:36.356949 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:36.356914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerStarted","Data":"4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759"} Jun 10 11:02:36.357431 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:36.356964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerStarted","Data":"18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7"} Jun 10 11:02:36.384485 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:36.384415 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podStartSLOduration=10.384390866 podStartE2EDuration="10.384390866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:02:36.383610191 +0000 UTC m=+1099.032666163" watchObservedRunningTime="2026-06-10 11:02:36.384390866 +0000 UTC m=+1099.033446839" Jun 10 11:02:37.042574 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:37.042533 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:37.042749 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:37.042615 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:39.041892 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:39.041851 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:39.042282 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:39.041928 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:39.043259 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:39.043219 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jun 10 11:02:39.043314 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:39.043244 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 10 11:02:42.042398 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:42.042350 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jun 10 11:02:42.047911 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:42.047880 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:42.051012 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:42.050988 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:45.042504 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:45.042463 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" Jun 10 11:02:48.042691 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:48.042657 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:48.043532 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:48.043508 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:02:52.733709 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.733665 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m"] Jun 10 11:02:52.739389 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.739365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.741709 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.741688 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-jklst\"" Jun 10 11:02:52.741838 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.741742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jun 10 11:02:52.741993 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.741977 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jun 10 11:02:52.742288 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.742270 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jun 10 11:02:52.754327 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.754305 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m"] Jun 10 11:02:52.841004 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.840973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52n2n\" (UniqueName: \"kubernetes.io/projected/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-kube-api-access-52n2n\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.841170 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.841016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.841170 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.841077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.841170 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.841112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-feast-data\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.941855 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.941814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-feast-data\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.942030 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.941875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52n2n\" (UniqueName: \"kubernetes.io/projected/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-kube-api-access-52n2n\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.942030 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.941914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.942030 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.941964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.942202 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:02:52.942084 2578 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Jun 10 11:02:52.942202 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:02:52.942155 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls podName:138aeaa4-9c84-4c96-8bfd-00a10d66cf44 nodeName:}" failed. No retries permitted until 2026-06-10 11:02:53.442134547 +0000 UTC m=+1116.091190503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls") pod "feast-driver-ranking-5bcc97f794-tq64m" (UID: "138aeaa4-9c84-4c96-8bfd-00a10d66cf44") : secret "feast-driver-ranking-registry-tls" not found Jun 10 11:02:52.942202 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.942190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-feast-data\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:52.942393 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:02:52.942302 2578 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Jun 10 11:02:52.942393 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:02:52.942361 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls podName:138aeaa4-9c84-4c96-8bfd-00a10d66cf44 nodeName:}" failed. No retries permitted until 2026-06-10 11:02:53.442344849 +0000 UTC m=+1116.091400815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls") pod "feast-driver-ranking-5bcc97f794-tq64m" (UID: "138aeaa4-9c84-4c96-8bfd-00a10d66cf44") : secret "feast-driver-ranking-online-tls" not found Jun 10 11:02:52.951961 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:52.951940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52n2n\" (UniqueName: \"kubernetes.io/projected/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-kube-api-access-52n2n\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:53.446832 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:53.446781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:53.447010 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:53.446871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:53.449272 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:53.449244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:53.449417 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:53.449396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls\") pod \"feast-driver-ranking-5bcc97f794-tq64m\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:53.650929 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:53.650893 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:02:53.798111 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:53.798085 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m"] Jun 10 11:02:53.800196 ip-10-0-138-231 kubenswrapper[2578]: W0610 11:02:53.800171 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138aeaa4_9c84_4c96_8bfd_00a10d66cf44.slice/crio-595e9028f03943fa58969ed48462b833ba63ea1f028aeab58991681207103ca1 WatchSource:0}: Error finding container 595e9028f03943fa58969ed48462b833ba63ea1f028aeab58991681207103ca1: Status 404 returned error can't find the container with id 595e9028f03943fa58969ed48462b833ba63ea1f028aeab58991681207103ca1 Jun 10 11:02:54.442177 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:54.442122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerStarted","Data":"21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3"} Jun 10 11:02:54.442377 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:54.442194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerStarted","Data":"595e9028f03943fa58969ed48462b833ba63ea1f028aeab58991681207103ca1"} Jun 10 11:02:58.459898 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:58.459861 2578 generic.go:356] "Generic (PLEG): container finished" podID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerID="21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3" exitCode=0 Jun 10 11:02:58.460280 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:58.459933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerDied","Data":"21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3"} Jun 10 11:02:59.466051 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:02:59.466011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerStarted","Data":"eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a"} Jun 10 11:03:05.494148 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:05.494115 2578 generic.go:356] "Generic (PLEG): container finished" podID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerID="eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a" exitCode=0 Jun 10 11:03:05.494496 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:05.494160 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerDied","Data":"eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a"} Jun 10 11:03:06.503108 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:06.503069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerStarted","Data":"f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e"} Jun 10 11:03:06.503473 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:06.503145 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerStarted","Data":"151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a"} Jun 10 11:03:06.527034 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:06.526972 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podStartSLOduration=14.526952211 podStartE2EDuration="14.526952211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:03:06.52560696 +0000 UTC m=+1129.174662933" watchObservedRunningTime="2026-06-10 11:03:06.526952211 +0000 UTC m=+1129.176008187" Jun 10 11:03:08.651865 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:08.651821 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:03:08.652234 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:08.651903 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:03:08.653361 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:08.653321 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 10 11:03:08.653361 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:08.653334 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Jun 10 11:03:11.651500 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:11.651449 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Jun 10 11:03:11.658661 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:11.658635 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:03:11.659158 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:11.659133 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:03:11.662843 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:11.662827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:03:13.651894 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:13.651848 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:03:14.651719 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:14.651677 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused" Jun 10 11:03:17.652530 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:17.652488 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:03:17.653269 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:03:17.653246 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:04:17.884126 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:17.884098 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 11:04:17.890044 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:17.890018 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 11:04:17.891548 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:17.891523 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:04:17.897005 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:17.896983 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:04:32.898586 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:32.898541 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw"] Jun 10 11:04:32.901093 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:32.898837 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" podUID="03430f26-f2de-420e-b1b4-997940b5fccc" containerName="manager" containerID="cri-o://5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819" gracePeriod=10 Jun 10 11:04:33.168451 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.168420 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 11:04:33.191424 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.191289 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2xct\" (UniqueName: \"kubernetes.io/projected/03430f26-f2de-420e-b1b4-997940b5fccc-kube-api-access-h2xct\") pod \"03430f26-f2de-420e-b1b4-997940b5fccc\" (UID: \"03430f26-f2de-420e-b1b4-997940b5fccc\") " Jun 10 11:04:33.194589 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.194548 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03430f26-f2de-420e-b1b4-997940b5fccc-kube-api-access-h2xct" (OuterVolumeSpecName: "kube-api-access-h2xct") pod "03430f26-f2de-420e-b1b4-997940b5fccc" (UID: "03430f26-f2de-420e-b1b4-997940b5fccc"). InnerVolumeSpecName "kube-api-access-h2xct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:04:33.231368 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.231334 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994"] Jun 10 11:04:33.231668 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.231602 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="registry" containerID="cri-o://18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7" gracePeriod=30 Jun 10 11:04:33.231838 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.231691 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" containerID="cri-o://4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759" gracePeriod=30 Jun 10 11:04:33.238167 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.238133 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m"] Jun 10 11:04:33.238441 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.238390 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="registry" containerID="cri-o://151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a" gracePeriod=30 Jun 10 11:04:33.238547 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.238445 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" containerID="cri-o://f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e" gracePeriod=30 Jun 10 11:04:33.292578 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.292534 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xct\" (UniqueName: \"kubernetes.io/projected/03430f26-f2de-420e-b1b4-997940b5fccc-kube-api-access-h2xct\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:04:33.910768 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.910726 2578 generic.go:356] "Generic (PLEG): container finished" podID="03430f26-f2de-420e-b1b4-997940b5fccc" containerID="5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819" exitCode=0 Jun 10 11:04:33.911236 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.910777 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" Jun 10 11:04:33.911236 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.910776 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" event={"ID":"03430f26-f2de-420e-b1b4-997940b5fccc","Type":"ContainerDied","Data":"5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819"} Jun 10 11:04:33.911236 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.910921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw" event={"ID":"03430f26-f2de-420e-b1b4-997940b5fccc","Type":"ContainerDied","Data":"16931c4b29ed5a32440e48148396c94b3b731376168423b08562d2fc36b84d98"} Jun 10 11:04:33.911236 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.910947 2578 scope.go:117] "RemoveContainer" containerID="5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819" Jun 10 11:04:33.921277 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.921247 2578 scope.go:117] "RemoveContainer" containerID="5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819" Jun 10 11:04:33.921570 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:04:33.921541 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819\": container with ID starting with 5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819 not found: ID does not exist" containerID="5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819" Jun 10 11:04:33.921638 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.921586 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819"} err="failed to get container status \"5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819\": rpc error: code = NotFound desc = could not find container \"5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819\": container with ID starting with 5124164548e2af64ca6ee6efb4687e2da040b82212b827e499f10d6030cf6819 not found: ID does not exist" Jun 10 11:04:33.928952 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.928915 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw"] Jun 10 11:04:33.932757 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:33.932729 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-69f8d7bdcc-djbkw"] Jun 10 11:04:34.918965 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:34.918925 2578 generic.go:356] "Generic (PLEG): container finished" podID="64b580db-adec-498b-a97f-26340d7f6b0d" containerID="4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759" exitCode=0 Jun 10 11:04:34.919428 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:34.918993 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerDied","Data":"4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759"} Jun 10 11:04:34.922876 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:34.922833 2578 generic.go:356] "Generic (PLEG): container finished" podID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerID="f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e" exitCode=0 Jun 10 11:04:34.922876 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:34.922873 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerDied","Data":"f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e"} Jun 10 11:04:35.027553 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.027519 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-txcrh"] Jun 10 11:04:35.027736 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.027699 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" podUID="af34b5d0-2b91-4a6a-918a-401d45f8cd2e" containerName="postgres" containerID="cri-o://0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4" gracePeriod=30 Jun 10 11:04:35.034460 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.034427 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4w4v7"] Jun 10 11:04:35.034652 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.034623 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" podUID="538e9571-c302-4a94-8278-49a67076dbed" containerName="redis" containerID="cri-o://99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c" gracePeriod=30 Jun 10 11:04:35.294579 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.294558 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:04:35.297320 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.297299 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" Jun 10 11:04:35.307427 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.307393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt6kq\" (UniqueName: \"kubernetes.io/projected/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-kube-api-access-zt6kq\") pod \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " Jun 10 11:04:35.307569 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.307438 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-postgresdata\") pod \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\" (UID: \"af34b5d0-2b91-4a6a-918a-401d45f8cd2e\") " Jun 10 11:04:35.309756 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.309727 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-kube-api-access-zt6kq" (OuterVolumeSpecName: "kube-api-access-zt6kq") pod "af34b5d0-2b91-4a6a-918a-401d45f8cd2e" (UID: "af34b5d0-2b91-4a6a-918a-401d45f8cd2e"). InnerVolumeSpecName "kube-api-access-zt6kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:04:35.338623 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.338587 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "af34b5d0-2b91-4a6a-918a-401d45f8cd2e" (UID: "af34b5d0-2b91-4a6a-918a-401d45f8cd2e"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:04:35.408505 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.408454 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9j8k\" (UniqueName: \"kubernetes.io/projected/538e9571-c302-4a94-8278-49a67076dbed-kube-api-access-w9j8k\") pod \"538e9571-c302-4a94-8278-49a67076dbed\" (UID: \"538e9571-c302-4a94-8278-49a67076dbed\") " Jun 10 11:04:35.408699 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.408686 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt6kq\" (UniqueName: \"kubernetes.io/projected/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-kube-api-access-zt6kq\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:04:35.408755 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.408708 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/af34b5d0-2b91-4a6a-918a-401d45f8cd2e-postgresdata\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:04:35.410626 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.410595 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538e9571-c302-4a94-8278-49a67076dbed-kube-api-access-w9j8k" (OuterVolumeSpecName: "kube-api-access-w9j8k") pod "538e9571-c302-4a94-8278-49a67076dbed" (UID: "538e9571-c302-4a94-8278-49a67076dbed"). InnerVolumeSpecName "kube-api-access-w9j8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:04:35.509948 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.509861 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9j8k\" (UniqueName: \"kubernetes.io/projected/538e9571-c302-4a94-8278-49a67076dbed-kube-api-access-w9j8k\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:04:35.853890 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.853857 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03430f26-f2de-420e-b1b4-997940b5fccc" path="/var/lib/kubelet/pods/03430f26-f2de-420e-b1b4-997940b5fccc/volumes" Jun 10 11:04:35.928719 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.928680 2578 generic.go:356] "Generic (PLEG): container finished" podID="538e9571-c302-4a94-8278-49a67076dbed" containerID="99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c" exitCode=0 Jun 10 11:04:35.928719 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.928722 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" Jun 10 11:04:35.929242 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.928776 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" event={"ID":"538e9571-c302-4a94-8278-49a67076dbed","Type":"ContainerDied","Data":"99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c"} Jun 10 11:04:35.929242 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.928830 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-4w4v7" event={"ID":"538e9571-c302-4a94-8278-49a67076dbed","Type":"ContainerDied","Data":"e23f2181be93e475a2d4d24c28eb2bc614dc1dcdb165d6af59c5bc94e25052c4"} Jun 10 11:04:35.929242 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.928850 2578 scope.go:117] "RemoveContainer" containerID="99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c" Jun 10 11:04:35.930194 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.930173 2578 generic.go:356] "Generic (PLEG): container finished" podID="af34b5d0-2b91-4a6a-918a-401d45f8cd2e" containerID="0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4" exitCode=0 Jun 10 11:04:35.930306 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.930209 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" event={"ID":"af34b5d0-2b91-4a6a-918a-401d45f8cd2e","Type":"ContainerDied","Data":"0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4"} Jun 10 11:04:35.930306 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.930217 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" Jun 10 11:04:35.930306 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.930236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-txcrh" event={"ID":"af34b5d0-2b91-4a6a-918a-401d45f8cd2e","Type":"ContainerDied","Data":"c06e8e6718fa072ede307d12f5c9982c1ae9ede7b6fd6938e53de5fcd6270c16"} Jun 10 11:04:35.939905 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.939853 2578 scope.go:117] "RemoveContainer" containerID="99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c" Jun 10 11:04:35.940327 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:04:35.940305 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c\": container with ID starting with 99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c not found: ID does not exist" containerID="99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c" Jun 10 11:04:35.940402 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.940335 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c"} err="failed to get container status \"99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c\": rpc error: code = NotFound desc = could not find container \"99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c\": container with ID starting with 99cd36fb38a09537620a52d664d52e5f4821e10ce68c46da9367718ec551e22c not found: ID does not exist" Jun 10 11:04:35.940402 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.940352 2578 scope.go:117] "RemoveContainer" containerID="0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4" Jun 10 11:04:35.944748 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.944723 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4w4v7"] Jun 10 11:04:35.948890 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.948871 2578 scope.go:117] "RemoveContainer" containerID="0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4" Jun 10 11:04:35.949133 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.949116 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-4w4v7"] Jun 10 11:04:35.949202 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:04:35.949156 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4\": container with ID starting with 0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4 not found: ID does not exist" containerID="0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4" Jun 10 11:04:35.949202 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.949177 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4"} err="failed to get container status \"0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4\": rpc error: code = NotFound desc = could not find container \"0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4\": container with ID starting with 0e58afb005018de0ab2ccbe8efcf663006d509dbe9663e94b0e20a769e7b4dd4 not found: ID does not exist" Jun 10 11:04:35.963760 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.963729 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-txcrh"] Jun 10 11:04:35.968206 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:35.968178 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-txcrh"] Jun 10 11:04:37.855391 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:37.855356 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="538e9571-c302-4a94-8278-49a67076dbed" path="/var/lib/kubelet/pods/538e9571-c302-4a94-8278-49a67076dbed/volumes" Jun 10 11:04:37.855763 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:37.855735 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af34b5d0-2b91-4a6a-918a-401d45f8cd2e" path="/var/lib/kubelet/pods/af34b5d0-2b91-4a6a-918a-401d45f8cd2e/volumes" Jun 10 11:04:41.659936 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:41.659894 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 10 11:04:42.049517 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:42.049415 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 10 11:04:51.659983 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:51.659930 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 10 11:04:52.049014 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:04:52.048921 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 10 11:05:01.659987 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:01.659939 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" Jun 10 11:05:01.660441 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:01.660106 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:05:02.048767 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:02.048672 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" Jun 10 11:05:02.048972 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:02.048817 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:05:03.908497 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:03.908446 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:05:03.912429 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:03.912407 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:05:04.043322 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043222 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64b580db-adec-498b-a97f-26340d7f6b0d-feast-data\") pod \"64b580db-adec-498b-a97f-26340d7f6b0d\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " Jun 10 11:05:04.043322 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043259 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-registry-tls\") pod \"64b580db-adec-498b-a97f-26340d7f6b0d\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " Jun 10 11:05:04.043322 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043292 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls\") pod \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " Jun 10 11:05:04.043322 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043317 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-feast-data\") pod \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " Jun 10 11:05:04.043667 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043342 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-online-tls\") pod \"64b580db-adec-498b-a97f-26340d7f6b0d\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " Jun 10 11:05:04.043667 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043364 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls\") pod \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " Jun 10 11:05:04.043667 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043391 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52n2n\" (UniqueName: \"kubernetes.io/projected/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-kube-api-access-52n2n\") pod \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\" (UID: \"138aeaa4-9c84-4c96-8bfd-00a10d66cf44\") " Jun 10 11:05:04.043667 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.043443 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64ttg\" (UniqueName: \"kubernetes.io/projected/64b580db-adec-498b-a97f-26340d7f6b0d-kube-api-access-64ttg\") pod \"64b580db-adec-498b-a97f-26340d7f6b0d\" (UID: \"64b580db-adec-498b-a97f-26340d7f6b0d\") " Jun 10 11:05:04.044354 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.044325 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-feast-data" (OuterVolumeSpecName: "feast-data") pod "138aeaa4-9c84-4c96-8bfd-00a10d66cf44" (UID: "138aeaa4-9c84-4c96-8bfd-00a10d66cf44"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:05:04.046191 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.046165 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "138aeaa4-9c84-4c96-8bfd-00a10d66cf44" (UID: "138aeaa4-9c84-4c96-8bfd-00a10d66cf44"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:05:04.046348 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.046316 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "64b580db-adec-498b-a97f-26340d7f6b0d" (UID: "64b580db-adec-498b-a97f-26340d7f6b0d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:05:04.046348 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.046323 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls" (OuterVolumeSpecName: "online-tls") pod "138aeaa4-9c84-4c96-8bfd-00a10d66cf44" (UID: "138aeaa4-9c84-4c96-8bfd-00a10d66cf44"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:05:04.046348 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.046337 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-online-tls" (OuterVolumeSpecName: "online-tls") pod "64b580db-adec-498b-a97f-26340d7f6b0d" (UID: "64b580db-adec-498b-a97f-26340d7f6b0d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:05:04.046542 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.046364 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b580db-adec-498b-a97f-26340d7f6b0d-kube-api-access-64ttg" (OuterVolumeSpecName: "kube-api-access-64ttg") pod "64b580db-adec-498b-a97f-26340d7f6b0d" (UID: "64b580db-adec-498b-a97f-26340d7f6b0d"). InnerVolumeSpecName "kube-api-access-64ttg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:05:04.046768 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.046742 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-kube-api-access-52n2n" (OuterVolumeSpecName: "kube-api-access-52n2n") pod "138aeaa4-9c84-4c96-8bfd-00a10d66cf44" (UID: "138aeaa4-9c84-4c96-8bfd-00a10d66cf44"). InnerVolumeSpecName "kube-api-access-52n2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:05:04.053356 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.053330 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b580db-adec-498b-a97f-26340d7f6b0d-feast-data" (OuterVolumeSpecName: "feast-data") pod "64b580db-adec-498b-a97f-26340d7f6b0d" (UID: "64b580db-adec-498b-a97f-26340d7f6b0d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:05:04.055672 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.055644 2578 generic.go:356] "Generic (PLEG): container finished" podID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerID="151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a" exitCode=137 Jun 10 11:05:04.055759 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.055708 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" Jun 10 11:05:04.055759 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.055717 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerDied","Data":"151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a"} Jun 10 11:05:04.055888 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.055769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m" event={"ID":"138aeaa4-9c84-4c96-8bfd-00a10d66cf44","Type":"ContainerDied","Data":"595e9028f03943fa58969ed48462b833ba63ea1f028aeab58991681207103ca1"} Jun 10 11:05:04.055888 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.055833 2578 scope.go:117] "RemoveContainer" containerID="f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e" Jun 10 11:05:04.058089 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.058063 2578 generic.go:356] "Generic (PLEG): container finished" podID="64b580db-adec-498b-a97f-26340d7f6b0d" containerID="18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7" exitCode=137 Jun 10 11:05:04.058189 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.058094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerDied","Data":"18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7"} Jun 10 11:05:04.058189 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.058112 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" Jun 10 11:05:04.058189 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.058140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994" event={"ID":"64b580db-adec-498b-a97f-26340d7f6b0d","Type":"ContainerDied","Data":"1e5e32b68ce8cb92bdf94f95c68027dd8f6b43087a9da9bef115b2fa08d7a739"} Jun 10 11:05:04.066215 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.066195 2578 scope.go:117] "RemoveContainer" containerID="151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a" Jun 10 11:05:04.074279 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.074263 2578 scope.go:117] "RemoveContainer" containerID="eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a" Jun 10 11:05:04.083345 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.083279 2578 scope.go:117] "RemoveContainer" containerID="21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3" Jun 10 11:05:04.085562 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.085542 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m"] Jun 10 11:05:04.091425 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.091408 2578 scope.go:117] "RemoveContainer" containerID="f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e" Jun 10 11:05:04.091705 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.091682 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e\": container with ID starting with f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e not found: ID does not exist" containerID="f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e" Jun 10 11:05:04.091762 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.091714 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e"} err="failed to get container status \"f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e\": rpc error: code = NotFound desc = could not find container \"f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e\": container with ID starting with f71f58f8b5a3b23b6223528cf9860271da94298de983f29d91ddd60aac2e254e not found: ID does not exist" Jun 10 11:05:04.091762 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.091731 2578 scope.go:117] "RemoveContainer" containerID="151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a" Jun 10 11:05:04.092022 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.091998 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a\": container with ID starting with 151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a not found: ID does not exist" containerID="151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a" Jun 10 11:05:04.092123 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.092025 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a"} err="failed to get container status \"151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a\": rpc error: code = NotFound desc = could not find container \"151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a\": container with ID starting with 151b5cfb02dbe8912df03e2f0b6660ad02da2cd228f3cf55daf5e591198d111a not found: ID does not exist" Jun 10 11:05:04.092123 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.092042 2578 scope.go:117] "RemoveContainer" containerID="eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a" Jun 10 11:05:04.092327 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.092311 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a\": container with ID starting with eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a not found: ID does not exist" containerID="eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a" Jun 10 11:05:04.092402 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.092331 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a"} err="failed to get container status \"eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a\": rpc error: code = NotFound desc = could not find container \"eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a\": container with ID starting with eb1c65d12d06b6ca464d6f998b8384abd104e87a93c5eddeb386b3747b13d37a not found: ID does not exist" Jun 10 11:05:04.092402 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.092378 2578 scope.go:117] "RemoveContainer" containerID="21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3" Jun 10 11:05:04.092550 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.092530 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5bcc97f794-tq64m"] Jun 10 11:05:04.092643 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.092628 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3\": container with ID starting with 21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3 not found: ID does not exist" containerID="21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3" Jun 10 11:05:04.092686 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.092648 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3"} err="failed to get container status \"21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3\": rpc error: code = NotFound desc = could not find container \"21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3\": container with ID starting with 21dd77954227a02f4436965ab3033d293129f1f4c5afbac3113ddb7fe4d81cc3 not found: ID does not exist" Jun 10 11:05:04.092686 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.092661 2578 scope.go:117] "RemoveContainer" containerID="4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759" Jun 10 11:05:04.100056 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.100039 2578 scope.go:117] "RemoveContainer" containerID="18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7" Jun 10 11:05:04.106363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.106341 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994"] Jun 10 11:05:04.108096 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.108080 2578 scope.go:117] "RemoveContainer" containerID="0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194" Jun 10 11:05:04.111734 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.111715 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6944f8477c-79994"] Jun 10 11:05:04.115492 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.115476 2578 scope.go:117] "RemoveContainer" containerID="b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87" Jun 10 11:05:04.122668 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.122654 2578 scope.go:117] "RemoveContainer" containerID="4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759" Jun 10 11:05:04.122970 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.122949 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759\": container with ID starting with 4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759 not found: ID does not exist" containerID="4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759" Jun 10 11:05:04.123057 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.122979 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759"} err="failed to get container status \"4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759\": rpc error: code = NotFound desc = could not find container \"4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759\": container with ID starting with 4393a3715cc75718f4a7ed5647fae568ae8ce65e6d28468f591a40ada849f759 not found: ID does not exist" Jun 10 11:05:04.123057 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.123002 2578 scope.go:117] "RemoveContainer" containerID="18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7" Jun 10 11:05:04.123265 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.123245 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7\": container with ID starting with 18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7 not found: ID does not exist" containerID="18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7" Jun 10 11:05:04.123304 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.123272 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7"} err="failed to get container status \"18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7\": rpc error: code = NotFound desc = could not find container \"18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7\": container with ID starting with 18eadd27c35cfcfa8e3ddd26f910b3736699f2273c9f8982bd726379787017b7 not found: ID does not exist" Jun 10 11:05:04.123304 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.123287 2578 scope.go:117] "RemoveContainer" containerID="0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194" Jun 10 11:05:04.123514 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.123500 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194\": container with ID starting with 0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194 not found: ID does not exist" containerID="0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194" Jun 10 11:05:04.123558 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.123517 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194"} err="failed to get container status \"0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194\": rpc error: code = NotFound desc = could not find container \"0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194\": container with ID starting with 0c6bb6a2408f5f740ff199076e224dc739158ab5e18c69ce1534e76ba6da9194 not found: ID does not exist" Jun 10 11:05:04.123558 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.123529 2578 scope.go:117] "RemoveContainer" containerID="b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87" Jun 10 11:05:04.123736 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:05:04.123719 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87\": container with ID starting with b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87 not found: ID does not exist" containerID="b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87" Jun 10 11:05:04.123788 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.123743 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87"} err="failed to get container status \"b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87\": rpc error: code = NotFound desc = could not find container \"b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87\": container with ID starting with b2487583d4c75129ed2529a2cacac94cb61e088e55512f1625f0893cb8fbbc87 not found: ID does not exist" Jun 10 11:05:04.144145 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144121 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:04.144145 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144143 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-feast-data\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:04.144264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144151 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-online-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:04.144264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144158 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-online-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:04.144264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144167 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52n2n\" (UniqueName: \"kubernetes.io/projected/138aeaa4-9c84-4c96-8bfd-00a10d66cf44-kube-api-access-52n2n\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:04.144264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144176 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ttg\" (UniqueName: \"kubernetes.io/projected/64b580db-adec-498b-a97f-26340d7f6b0d-kube-api-access-64ttg\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:04.144264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144184 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/64b580db-adec-498b-a97f-26340d7f6b0d-feast-data\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:04.144264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:04.144192 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/64b580db-adec-498b-a97f-26340d7f6b0d-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:05:05.854787 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:05.854714 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" path="/var/lib/kubelet/pods/138aeaa4-9c84-4c96-8bfd-00a10d66cf44/volumes" Jun 10 11:05:05.855258 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:05:05.855242 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" path="/var/lib/kubelet/pods/64b580db-adec-498b-a97f-26340d7f6b0d/volumes" Jun 10 11:09:17.909934 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:09:17.909903 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 11:09:17.916438 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:09:17.916414 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 11:09:17.916707 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:09:17.916689 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:09:17.923102 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:09:17.923082 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:14:17.934470 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:14:17.934434 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 11:14:17.941195 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:14:17.941170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:14:17.941671 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:14:17.941653 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 11:14:17.948186 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:14:17.948169 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:15:29.969904 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.969866 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c9tlv/must-gather-vm5bj"] Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970173 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538e9571-c302-4a94-8278-49a67076dbed" containerName="redis" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970184 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="538e9571-c302-4a94-8278-49a67076dbed" containerName="redis" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970196 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="feast-init" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970201 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="feast-init" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970211 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03430f26-f2de-420e-b1b4-997940b5fccc" containerName="manager" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970216 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03430f26-f2de-420e-b1b4-997940b5fccc" containerName="manager" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970222 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af34b5d0-2b91-4a6a-918a-401d45f8cd2e" containerName="postgres" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970227 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="af34b5d0-2b91-4a6a-918a-401d45f8cd2e" containerName="postgres" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970234 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970240 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970245 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="feast-apply" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970250 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="feast-apply" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970262 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="feast-init" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970268 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="feast-init" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970274 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="registry" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970279 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="registry" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970290 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="registry" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970295 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="registry" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970303 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970307 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970313 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="feast-apply" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970318 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="feast-apply" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970363 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="af34b5d0-2b91-4a6a-918a-401d45f8cd2e" containerName="postgres" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970371 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="538e9571-c302-4a94-8278-49a67076dbed" containerName="redis" Jun 10 11:15:29.970363 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970376 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="registry" Jun 10 11:15:29.971126 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970381 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b580db-adec-498b-a97f-26340d7f6b0d" containerName="online" Jun 10 11:15:29.971126 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970386 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="online" Jun 10 11:15:29.971126 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970393 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="138aeaa4-9c84-4c96-8bfd-00a10d66cf44" containerName="registry" Jun 10 11:15:29.971126 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.970399 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03430f26-f2de-420e-b1b4-997940b5fccc" containerName="manager" Jun 10 11:15:29.973368 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.973352 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:29.975405 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.975378 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9tlv\"/\"kube-root-ca.crt\"" Jun 10 11:15:29.975591 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.975378 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9tlv\"/\"openshift-service-ca.crt\"" Jun 10 11:15:29.975591 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.975417 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c9tlv\"/\"default-dockercfg-j7bzf\"" Jun 10 11:15:29.984901 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.984876 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9tlv/must-gather-vm5bj"] Jun 10 11:15:29.994424 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.994379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d610c84f-615c-4722-a35a-12d511c49b2e-must-gather-output\") pod \"must-gather-vm5bj\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:29.994569 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:29.994454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ddc\" (UniqueName: \"kubernetes.io/projected/d610c84f-615c-4722-a35a-12d511c49b2e-kube-api-access-f2ddc\") pod \"must-gather-vm5bj\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:30.095446 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.095402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d610c84f-615c-4722-a35a-12d511c49b2e-must-gather-output\") pod \"must-gather-vm5bj\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:30.095658 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.095462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ddc\" (UniqueName: \"kubernetes.io/projected/d610c84f-615c-4722-a35a-12d511c49b2e-kube-api-access-f2ddc\") pod \"must-gather-vm5bj\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:30.095831 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.095775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d610c84f-615c-4722-a35a-12d511c49b2e-must-gather-output\") pod \"must-gather-vm5bj\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:30.103686 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.103660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ddc\" (UniqueName: \"kubernetes.io/projected/d610c84f-615c-4722-a35a-12d511c49b2e-kube-api-access-f2ddc\") pod \"must-gather-vm5bj\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:30.300647 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.300563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:30.433010 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.432984 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9tlv/must-gather-vm5bj"] Jun 10 11:15:30.435159 ip-10-0-138-231 kubenswrapper[2578]: W0610 11:15:30.435130 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd610c84f_615c_4722_a35a_12d511c49b2e.slice/crio-f59466d0952f515b6dd7b5d20cfcf82b37851a1cd46055c901466e39529dddf0 WatchSource:0}: Error finding container f59466d0952f515b6dd7b5d20cfcf82b37851a1cd46055c901466e39529dddf0: Status 404 returned error can't find the container with id f59466d0952f515b6dd7b5d20cfcf82b37851a1cd46055c901466e39529dddf0 Jun 10 11:15:30.437150 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.437133 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 10 11:15:30.518546 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:30.518508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" event={"ID":"d610c84f-615c-4722-a35a-12d511c49b2e","Type":"ContainerStarted","Data":"f59466d0952f515b6dd7b5d20cfcf82b37851a1cd46055c901466e39529dddf0"} Jun 10 11:15:37.553221 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:37.553181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" event={"ID":"d610c84f-615c-4722-a35a-12d511c49b2e","Type":"ContainerStarted","Data":"148c231248e0381413522ffcd3e78203a96bb103fb469cb770cb9aec0b9ccaf6"} Jun 10 11:15:37.553221 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:37.553224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" event={"ID":"d610c84f-615c-4722-a35a-12d511c49b2e","Type":"ContainerStarted","Data":"48c9649b03822dfebda6a863fcde38ea1d9ccc8fca539090349c63bb5c7bab7e"} Jun 10 11:15:37.572872 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:37.572817 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" podStartSLOduration=2.5505918579999998 podStartE2EDuration="8.572782364s" totalImagesPullingTime="6.022190506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:15:37.572346501 +0000 UTC m=+1880.221402474" watchObservedRunningTime="2026-06-10 11:15:37.572782364 +0000 UTC m=+1880.221838336" Jun 10 11:15:46.590562 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:46.590526 2578 generic.go:356] "Generic (PLEG): container finished" podID="d610c84f-615c-4722-a35a-12d511c49b2e" containerID="48c9649b03822dfebda6a863fcde38ea1d9ccc8fca539090349c63bb5c7bab7e" exitCode=0 Jun 10 11:15:46.591025 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:46.590619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" event={"ID":"d610c84f-615c-4722-a35a-12d511c49b2e","Type":"ContainerDied","Data":"48c9649b03822dfebda6a863fcde38ea1d9ccc8fca539090349c63bb5c7bab7e"} Jun 10 11:15:46.591070 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:46.591031 2578 scope.go:117] "RemoveContainer" containerID="48c9649b03822dfebda6a863fcde38ea1d9ccc8fca539090349c63bb5c7bab7e" Jun 10 11:15:47.113702 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.113672 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c9tlv_must-gather-vm5bj_d610c84f-615c-4722-a35a-12d511c49b2e/gather/0.log" Jun 10 11:15:47.610649 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.610614 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g7cc7/must-gather-fjhhs"] Jun 10 11:15:47.613690 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.613673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:47.615822 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.615775 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g7cc7\"/\"default-dockercfg-8fnkx\"" Jun 10 11:15:47.616246 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.616229 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g7cc7\"/\"kube-root-ca.crt\"" Jun 10 11:15:47.616877 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.616861 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g7cc7\"/\"openshift-service-ca.crt\"" Jun 10 11:15:47.623603 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.623580 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7cc7/must-gather-fjhhs"] Jun 10 11:15:47.742476 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.742439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57185b88-c8b9-4ff5-8f8d-67e5c6b541c5-must-gather-output\") pod \"must-gather-fjhhs\" (UID: \"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5\") " pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:47.742476 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.742479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6sb\" (UniqueName: \"kubernetes.io/projected/57185b88-c8b9-4ff5-8f8d-67e5c6b541c5-kube-api-access-kk6sb\") pod \"must-gather-fjhhs\" (UID: \"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5\") " pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:47.843711 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.843680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57185b88-c8b9-4ff5-8f8d-67e5c6b541c5-must-gather-output\") pod \"must-gather-fjhhs\" (UID: \"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5\") " pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:47.843909 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.843716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6sb\" (UniqueName: \"kubernetes.io/projected/57185b88-c8b9-4ff5-8f8d-67e5c6b541c5-kube-api-access-kk6sb\") pod \"must-gather-fjhhs\" (UID: \"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5\") " pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:47.844059 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.844038 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57185b88-c8b9-4ff5-8f8d-67e5c6b541c5-must-gather-output\") pod \"must-gather-fjhhs\" (UID: \"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5\") " pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:47.852055 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.852030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6sb\" (UniqueName: \"kubernetes.io/projected/57185b88-c8b9-4ff5-8f8d-67e5c6b541c5-kube-api-access-kk6sb\") pod \"must-gather-fjhhs\" (UID: \"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5\") " pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:47.922529 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:47.922426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7cc7/must-gather-fjhhs" Jun 10 11:15:48.064938 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:48.064911 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7cc7/must-gather-fjhhs"] Jun 10 11:15:48.067198 ip-10-0-138-231 kubenswrapper[2578]: W0610 11:15:48.067171 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57185b88_c8b9_4ff5_8f8d_67e5c6b541c5.slice/crio-e4afe5ce55514935ec3aac3017975a89bc65651698580eae101ee4d5e270b077 WatchSource:0}: Error finding container e4afe5ce55514935ec3aac3017975a89bc65651698580eae101ee4d5e270b077: Status 404 returned error can't find the container with id e4afe5ce55514935ec3aac3017975a89bc65651698580eae101ee4d5e270b077 Jun 10 11:15:48.599710 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:48.599676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7cc7/must-gather-fjhhs" event={"ID":"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5","Type":"ContainerStarted","Data":"e4afe5ce55514935ec3aac3017975a89bc65651698580eae101ee4d5e270b077"} Jun 10 11:15:49.605891 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:49.605845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7cc7/must-gather-fjhhs" event={"ID":"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5","Type":"ContainerStarted","Data":"ab13b1ffaa1ccf045c8a8a0265a1784064ec4bbacf69e1ec78b065c0147f2243"} Jun 10 11:15:49.606264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:49.605914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7cc7/must-gather-fjhhs" event={"ID":"57185b88-c8b9-4ff5-8f8d-67e5c6b541c5","Type":"ContainerStarted","Data":"ddcd58923511039a39422decb40c29b0220d952e6a3daf33042f635c5979fcb2"} Jun 10 11:15:49.624216 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:49.624163 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g7cc7/must-gather-fjhhs" podStartSLOduration=1.824784745 podStartE2EDuration="2.624146098s" totalImagesPullingTime="799.361353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:15:49.622976226 +0000 UTC m=+1892.272032198" watchObservedRunningTime="2026-06-10 11:15:49.624146098 +0000 UTC m=+1892.273202072" Jun 10 11:15:50.340635 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:50.340563 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ptw8k_7f2862d8-beef-4218-b972-896923c4709a/global-pull-secret-syncer/0.log" Jun 10 11:15:50.418749 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:50.418719 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m86dg_a97c9c3b-0b82-410b-b415-b62f308b297b/konnectivity-agent/0.log" Jun 10 11:15:50.504720 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:50.504677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-231.ec2.internal_27887576b5ee87287ccccd6eae7bb07b/haproxy/0.log" Jun 10 11:15:52.453338 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.453293 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-c9tlv/must-gather-vm5bj"] Jun 10 11:15:52.454293 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.454256 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" containerName="copy" containerID="cri-o://148c231248e0381413522ffcd3e78203a96bb103fb469cb770cb9aec0b9ccaf6" gracePeriod=2 Jun 10 11:15:52.480455 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.480421 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-c9tlv/must-gather-vm5bj"] Jun 10 11:15:52.629459 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.629381 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c9tlv_must-gather-vm5bj_d610c84f-615c-4722-a35a-12d511c49b2e/copy/0.log" Jun 10 11:15:52.630108 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.629990 2578 generic.go:356] "Generic (PLEG): container finished" podID="d610c84f-615c-4722-a35a-12d511c49b2e" containerID="148c231248e0381413522ffcd3e78203a96bb103fb469cb770cb9aec0b9ccaf6" exitCode=143 Jun 10 11:15:52.904264 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.903764 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c9tlv_must-gather-vm5bj_d610c84f-615c-4722-a35a-12d511c49b2e/copy/0.log" Jun 10 11:15:52.909984 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.909723 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:52.911750 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:15:52.911575 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vm5bj\" is forbidden: User \"system:node:ip-10-0-138-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c9tlv\": no relationship found between node 'ip-10-0-138-231.ec2.internal' and this object" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:52.988747 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.987766 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2ddc\" (UniqueName: \"kubernetes.io/projected/d610c84f-615c-4722-a35a-12d511c49b2e-kube-api-access-f2ddc\") pod \"d610c84f-615c-4722-a35a-12d511c49b2e\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " Jun 10 11:15:52.988747 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.987853 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d610c84f-615c-4722-a35a-12d511c49b2e-must-gather-output\") pod \"d610c84f-615c-4722-a35a-12d511c49b2e\" (UID: \"d610c84f-615c-4722-a35a-12d511c49b2e\") " Jun 10 11:15:52.990057 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.990033 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d610c84f-615c-4722-a35a-12d511c49b2e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d610c84f-615c-4722-a35a-12d511c49b2e" (UID: "d610c84f-615c-4722-a35a-12d511c49b2e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:15:52.991245 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:52.991222 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d610c84f-615c-4722-a35a-12d511c49b2e-kube-api-access-f2ddc" (OuterVolumeSpecName: "kube-api-access-f2ddc") pod "d610c84f-615c-4722-a35a-12d511c49b2e" (UID: "d610c84f-615c-4722-a35a-12d511c49b2e"). InnerVolumeSpecName "kube-api-access-f2ddc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:15:53.089489 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:53.089459 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2ddc\" (UniqueName: \"kubernetes.io/projected/d610c84f-615c-4722-a35a-12d511c49b2e-kube-api-access-f2ddc\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:15:53.089712 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:53.089698 2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d610c84f-615c-4722-a35a-12d511c49b2e-must-gather-output\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\"" Jun 10 11:15:53.637683 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:53.637654 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c9tlv_must-gather-vm5bj_d610c84f-615c-4722-a35a-12d511c49b2e/copy/0.log" Jun 10 11:15:53.638608 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:53.638590 2578 scope.go:117] "RemoveContainer" containerID="148c231248e0381413522ffcd3e78203a96bb103fb469cb770cb9aec0b9ccaf6" Jun 10 11:15:53.638924 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:53.638911 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:53.652315 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:15:53.652023 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vm5bj\" is forbidden: User \"system:node:ip-10-0-138-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c9tlv\": no relationship found between node 'ip-10-0-138-231.ec2.internal' and this object" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:53.655100 ip-10-0-138-231 kubenswrapper[2578]: E0610 11:15:53.655071 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vm5bj\" is forbidden: User \"system:node:ip-10-0-138-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c9tlv\": no relationship found between node 'ip-10-0-138-231.ec2.internal' and this object" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" pod="openshift-must-gather-c9tlv/must-gather-vm5bj" Jun 10 11:15:53.669102 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:53.667919 2578 scope.go:117] "RemoveContainer" containerID="48c9649b03822dfebda6a863fcde38ea1d9ccc8fca539090349c63bb5c7bab7e" Jun 10 11:15:53.858315 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:53.858264 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" path="/var/lib/kubelet/pods/d610c84f-615c-4722-a35a-12d511c49b2e/volumes" Jun 10 11:15:54.559497 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:54.559459 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-74bbf69bbb-4gmkb_20f42b9d-d00f-4058-b109-072b242bc4e0/cluster-monitoring-operator/0.log" Jun 10 11:15:54.591550 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:54.591521 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-57bbf8bfb5-t2nx9_d9053fec-1dbf-47ea-988f-ac88247ba7f2/kube-state-metrics/0.log" Jun 10 11:15:54.617698 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:54.617672 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-57bbf8bfb5-t2nx9_d9053fec-1dbf-47ea-988f-ac88247ba7f2/kube-rbac-proxy-main/0.log" Jun 10 11:15:54.652260 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:54.652225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-57bbf8bfb5-t2nx9_d9053fec-1dbf-47ea-988f-ac88247ba7f2/kube-rbac-proxy-self/0.log" Jun 10 11:15:54.942144 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:54.942052 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zw5x6_66fdcddc-54e9-4375-9b6b-799eec109808/node-exporter/0.log" Jun 10 11:15:54.968019 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:54.967988 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zw5x6_66fdcddc-54e9-4375-9b6b-799eec109808/kube-rbac-proxy/0.log" Jun 10 11:15:54.993904 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:54.993876 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zw5x6_66fdcddc-54e9-4375-9b6b-799eec109808/init-textfile/0.log" Jun 10 11:15:55.032218 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.032173 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-65f78d5c66-pnzqc_1a190f80-8f1f-4c2b-b914-358130b54f15/kube-rbac-proxy-main/0.log" Jun 10 11:15:55.061139 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.061100 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-65f78d5c66-pnzqc_1a190f80-8f1f-4c2b-b914-358130b54f15/kube-rbac-proxy-self/0.log" Jun 10 11:15:55.085564 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.085523 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-65f78d5c66-pnzqc_1a190f80-8f1f-4c2b-b914-358130b54f15/openshift-state-metrics/0.log" Jun 10 11:15:55.322108 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.322077 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7f7d445d84-tpcpw_5f3f3363-ad5e-4309-8eb4-d54c7479b263/prometheus-operator/0.log" Jun 10 11:15:55.346716 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.346681 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7f7d445d84-tpcpw_5f3f3363-ad5e-4309-8eb4-d54c7479b263/kube-rbac-proxy/0.log" Jun 10 11:15:55.382072 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.382041 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5b6b8f594-s6jfk_d63ff657-91a0-4b96-ad25-87088038b125/prometheus-operator-admission-webhook/0.log" Jun 10 11:15:55.555140 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.555106 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8544f65558-rv8l6_e7d6ed81-7464-47bf-9ce7-9e1c59d3e738/thanos-query/0.log" Jun 10 11:15:55.592019 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.591935 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8544f65558-rv8l6_e7d6ed81-7464-47bf-9ce7-9e1c59d3e738/kube-rbac-proxy-web/0.log" Jun 10 11:15:55.625052 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.625017 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8544f65558-rv8l6_e7d6ed81-7464-47bf-9ce7-9e1c59d3e738/kube-rbac-proxy/0.log" Jun 10 11:15:55.660437 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.660402 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8544f65558-rv8l6_e7d6ed81-7464-47bf-9ce7-9e1c59d3e738/prom-label-proxy/0.log" Jun 10 11:15:55.691144 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.690314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8544f65558-rv8l6_e7d6ed81-7464-47bf-9ce7-9e1c59d3e738/kube-rbac-proxy-rules/0.log" Jun 10 11:15:55.720089 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:55.720065 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8544f65558-rv8l6_e7d6ed81-7464-47bf-9ce7-9e1c59d3e738/kube-rbac-proxy-metrics/0.log" Jun 10 11:15:57.265354 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.265324 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/1.log" Jun 10 11:15:57.271724 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.271698 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-668fdbf85c-9q6dc_d7c4513f-e58c-41c5-bda8-f64927fd7358/console-operator/2.log" Jun 10 11:15:57.646735 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.646698 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c"] Jun 10 11:15:57.647055 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.647040 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" containerName="copy" Jun 10 11:15:57.647055 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.647055 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" containerName="copy" Jun 10 11:15:57.647190 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.647073 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" containerName="gather" Jun 10 11:15:57.647190 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.647079 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" containerName="gather" Jun 10 11:15:57.647190 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.647131 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" containerName="gather" Jun 10 11:15:57.647190 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.647140 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d610c84f-615c-4722-a35a-12d511c49b2e" containerName="copy" Jun 10 11:15:57.652452 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.652423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.665702 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.665676 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c"] Jun 10 11:15:57.731138 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.731105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-sys\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.731310 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.731150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-proc\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.731310 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.731200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-lib-modules\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.731310 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.731241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5r2\" (UniqueName: \"kubernetes.io/projected/c6b9e8db-7af8-4b1b-8972-9650b170f516-kube-api-access-hb5r2\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.731310 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.731299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-podres\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.832124 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-podres\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.832372 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-sys\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.832512 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-proc\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.832701 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-lib-modules\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.833049 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.833027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5r2\" (UniqueName: \"kubernetes.io/projected/c6b9e8db-7af8-4b1b-8972-9650b170f516-kube-api-access-hb5r2\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.833415 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832642 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-proc\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.833511 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-lib-modules\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.833511 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-sys\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.833511 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.832353 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6b9e8db-7af8-4b1b-8972-9650b170f516-podres\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.842113 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.842088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5r2\" (UniqueName: \"kubernetes.io/projected/c6b9e8db-7af8-4b1b-8972-9650b170f516-kube-api-access-hb5r2\") pod \"perf-node-gather-daemonset-skr6c\" (UID: \"c6b9e8db-7af8-4b1b-8972-9650b170f516\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:57.967023 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:57.966938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:58.134468 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:58.134191 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c"] Jun 10 11:15:58.137193 ip-10-0-138-231 kubenswrapper[2578]: W0610 11:15:58.137150 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6b9e8db_7af8_4b1b_8972_9650b170f516.slice/crio-2cf6325e9eb58287d0b97781d05d563dfe42c9f5c4541e50538da99dca36fcaf WatchSource:0}: Error finding container 2cf6325e9eb58287d0b97781d05d563dfe42c9f5c4541e50538da99dca36fcaf: Status 404 returned error can't find the container with id 2cf6325e9eb58287d0b97781d05d563dfe42c9f5c4541e50538da99dca36fcaf Jun 10 11:15:58.677607 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:58.677575 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" event={"ID":"c6b9e8db-7af8-4b1b-8972-9650b170f516","Type":"ContainerStarted","Data":"9c08dbd0f98ec2bdf1c28cbf074318e6821aeadacf22ecfe2cb1458c82ebfd9e"} Jun 10 11:15:58.678080 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:58.677626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" event={"ID":"c6b9e8db-7af8-4b1b-8972-9650b170f516","Type":"ContainerStarted","Data":"2cf6325e9eb58287d0b97781d05d563dfe42c9f5c4541e50538da99dca36fcaf"} Jun 10 11:15:58.678080 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:58.677648 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:15:58.706028 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:58.705970 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" podStartSLOduration=1.705950608 podStartE2EDuration="1.705950608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 11:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:15:58.702751278 +0000 UTC m=+1901.351807262" watchObservedRunningTime="2026-06-10 11:15:58.705950608 +0000 UTC m=+1901.355006582" Jun 10 11:15:59.317633 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:59.317600 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vg7kl_cacd2e06-fcd6-43b0-aff9-6f114eddd501/dns/0.log" Jun 10 11:15:59.344667 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:59.344631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vg7kl_cacd2e06-fcd6-43b0-aff9-6f114eddd501/kube-rbac-proxy/0.log" Jun 10 11:15:59.372337 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:59.372316 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-29rcg_6bffc05c-b337-483c-8b1f-907780f2d0d2/dns-node-resolver/0.log" Jun 10 11:15:59.928759 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:15:59.928726 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ps8th_12b531fb-fb79-47d1-adab-9eca526c52de/node-ca/0.log" Jun 10 11:16:01.037393 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:01.037353 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42gfm_125476fa-5f69-45a5-8232-31541c6be892/serve-healthcheck-canary/0.log" Jun 10 11:16:01.482816 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:01.482772 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-896549976-rjfdv_dfc1a540-76c8-4702-be98-ba0d4a1759f3/insights-operator/0.log" Jun 10 11:16:01.483322 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:01.483303 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-896549976-rjfdv_dfc1a540-76c8-4702-be98-ba0d4a1759f3/insights-operator/1.log" Jun 10 11:16:01.507558 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:01.507535 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7c8x_88b447cf-9b8e-4774-b015-f23fd9426eba/kube-rbac-proxy/0.log" Jun 10 11:16:01.536152 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:01.536130 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7c8x_88b447cf-9b8e-4774-b015-f23fd9426eba/exporter/0.log" Jun 10 11:16:01.564643 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:01.564620 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7c8x_88b447cf-9b8e-4774-b015-f23fd9426eba/extractor/0.log" Jun 10 11:16:04.692881 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:04.692857 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-skr6c" Jun 10 11:16:06.038329 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:06.038299 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-955ffdf5f-8d2zc_53b6242f-c082-471c-8238-e57e35a3d780/migrator/0.log" Jun 10 11:16:06.099060 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:06.099028 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-955ffdf5f-8d2zc_53b6242f-c082-471c-8238-e57e35a3d780/graceful-termination/0.log" Jun 10 11:16:06.528410 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:06.528373 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55bb7c4d49-pszhc_4075f9cd-34d0-4601-b9ed-227e3918826e/kube-storage-version-migrator-operator/1.log" Jun 10 11:16:06.529452 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:06.529422 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55bb7c4d49-pszhc_4075f9cd-34d0-4601-b9ed-227e3918826e/kube-storage-version-migrator-operator/0.log" Jun 10 11:16:07.581414 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:07.581344 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4lzjt_5008bbcc-daab-4622-819f-bf83f17c50fe/kube-multus/0.log" Jun 10 11:16:08.071987 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.071953 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mh7hw_29011c56-f7dc-4418-8881-1cdfc9f277f1/kube-multus-additional-cni-plugins/0.log" Jun 10 11:16:08.109695 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.109668 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mh7hw_29011c56-f7dc-4418-8881-1cdfc9f277f1/egress-router-binary-copy/0.log" Jun 10 11:16:08.149536 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.149505 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mh7hw_29011c56-f7dc-4418-8881-1cdfc9f277f1/cni-plugins/0.log" Jun 10 11:16:08.181540 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.181514 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mh7hw_29011c56-f7dc-4418-8881-1cdfc9f277f1/bond-cni-plugin/0.log" Jun 10 11:16:08.208424 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.208400 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mh7hw_29011c56-f7dc-4418-8881-1cdfc9f277f1/routeoverride-cni/0.log" Jun 10 11:16:08.236081 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.236056 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mh7hw_29011c56-f7dc-4418-8881-1cdfc9f277f1/whereabouts-cni-bincopy/0.log" Jun 10 11:16:08.266592 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.266563 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mh7hw_29011c56-f7dc-4418-8881-1cdfc9f277f1/whereabouts-cni/0.log" Jun 10 11:16:08.428443 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.428358 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8t72g_8327523a-eb26-4c9d-847a-761b868b2d27/network-metrics-daemon/0.log" Jun 10 11:16:08.457382 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:08.457317 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8t72g_8327523a-eb26-4c9d-847a-761b868b2d27/kube-rbac-proxy/0.log" Jun 10 11:16:09.288684 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.288659 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-controller/0.log" Jun 10 11:16:09.314834 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.314793 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/1.log" Jun 10 11:16:09.315023 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.315008 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovn-acl-logging/0.log" Jun 10 11:16:09.338777 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.338753 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/kube-rbac-proxy-node/0.log" Jun 10 11:16:09.364439 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.364414 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/kube-rbac-proxy-ovn-metrics/0.log" Jun 10 11:16:09.389615 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.389586 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/northd/0.log" Jun 10 11:16:09.422267 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.422249 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/nbdb/0.log" Jun 10 11:16:09.449036 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.449006 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/sbdb/0.log" Jun 10 11:16:09.499971 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:09.499948 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s2n5_30c39d51-86a4-424a-97f0-f9c2c568f8db/ovnkube-controller/0.log" Jun 10 11:16:11.045904 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:11.045876 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-dbd9cf844-9sl4x_2e27c040-9441-4b68-b199-17cf79ee51d1/check-endpoints/0.log" Jun 10 11:16:11.076336 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:11.076309 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ccsb6_bc6fff07-0a5d-4065-8912-789624912795/network-check-target-container/0.log" Jun 10 11:16:12.220356 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:12.220324 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zwzdb_16d98a46-dd9c-47d4-aa38-47e65d1a3342/iptables-alerter/0.log" Jun 10 11:16:12.946130 ip-10-0-138-231 kubenswrapper[2578]: I0610 11:16:12.946098 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-l8pk6_b73c7432-8e5b-49d3-a90b-57a8fdeccce8/tuned/0.log"